Monitoring electricity consumption in the home is an important way to help reduce energy usage and Non-Intrusive Load Monitoring (NILM) techniques are a promising approach to obtain estimates of the electrical power consumption of individual appliances from aggregate measurements of voltage and/or current in the distribution system. In this paper, we discuss event detection algorithms used in the NILM literature and propose new metrics for evaluating them. In particular, we introduce metrics that incorporate information contained in the power signal instead of strict detection rates. We show that this information is important for NILM applications with the goal of improving appliance energy disaggregation. Our work was carried out on a week-long dataset of real residential power usage which we intend to make publicly available.