Downsampling

Overview

Data associated with a building in SCAN is aligned to a timestep which is a number of minutes that is a whole fraction of an hour ( 1, 2, 3, 4, 5, 6, 10, 12, 15, 20, 30, 60 ). This timestep is the building sample period. Of the available timesteps, ( 6, 10, 30, 60 ) are available as reporting intervals for Apache. When comparing SCAN data to Apache results, use the same interval. Changing the building sample period changes the expected window for channel data, so will cause recalculation of data quality metrics for all channels in the building.

Interpolation, plotting, expression evaluation and export of data to VE all use the building sample period as their time base. Each channel can also have its own expected sample period, which if not set will default to the building sample period. The channel's expected sample period is used to determine when to interpolate - gaps of less than five expected sample periods are interpolated - and for calculating the channel's population - the number of expected sample periods which have some data reported in them.

Data imported into SCAN does not necessarily align to the building sample period and will therefore need either downsampling or interpolating.

The way in which downsampling affects the data depends on the channel's sample type, and the total mode of the channel's value type.

Timestamps and slots

A raw data point has a timestamp and a value. A sequence of values as returned by SCAN consists of a sequence of time slots and the corresponding values. For example, in the examples below there is raw data with the following timestamps and values

time	00:01	00:07	00:17	00:27	00:37	01:45
value	   12       4      14       4       4       6

The building sample period is 15 minutes below. The values for eight time slots are requested, from 00:00 inclusive to 02:00 exclusive.

Raw data

SCAN stores the raw data at one minute resolution. When raw data is requested, the returned values are realigned to the building sample period. The value returned for each time slot is the raw data point whose timestamp is not greater than the start time of the slot.

Raw data result ( all sample types and total modes )

time	00:00	00:15	00:30	00:45	01:00	01:15	01:30	01:45   02:00	
value	    -       4       4       4       -       -       -       6       -   

Derived data

In SCAN, derived data is data derived from the raw data or by applying an expression to the values of other channels or other entities in the data model.

Interpolation may also be applied to the derived data. If interpolation is requested, then the value of a expression used for the is based on the interpolated data from other channels; otherwise the value of an expression is based on the derived data for other channels.

The derived data is modified to allow comparison with Apache results. What is done to the data depends on the channel's Sample Type flag, and the total mode of the value type of the channel's units.

Sample type

The following sample types are available, which are used to tell SCAN how to pre-process the stream of raw values:

Spot Value

Spot value samples are assumed to be the value of the variable at the given time stamp. When processed for comparison with Apache results, they are averaged over the time slot using linear interpolation. Gaps greater than the maximum gap period are not interpolated.

Discrete Stepping

Discrete stepping samples are used where the value only reports changes. The value is held constant until the next sample point. The maximum gap period has no effect, nor does the total mode of the channel units.

Average

This indicates the samples are already averaged. Downsampling will attempt to keep the value closest to the original and not apply further averaging which would smooth the values. The value returned will be the average of all samples in the time slot. The averaging is not time dependent. This is used for data sources which have already been processed, such as weather data services.

Cumulative

This indicates the sample value is a cumulative meter or similar output. The value the channel takes is the rate of change, which is the difference between readings divided by the time between the readings. For totalled or untotalled values, the average rate is used. For independent values, the total of positive changes is used.

Metered Value

This indicates the sample value is a total meter or similar output. The value recorded is the total of the channel since the last record, so the value the channel takes is the rate of use of the metered value. For totalled or untotalled values, the average rate of the meter is used. For independent values, the total of the metered value is used.

For both metered and cumualative values, as the channel's value is the result of dividing the raw data over a period, the first raw value is used only to determine the period to use for the second value. If a channel value for the period the the first value represents is desired, then add an appropriate value before it to tell SCAN when the period should start.

Total mode

The total mode is associated with the channel's units.

Untotalled

This is used for units such as temperature, where it is not meaningful to total the value over time. If multiple sample points occur in a time slot, the value is averaged over the time slot. The total value is replaced by the mean. It is not really meaningful when combined with cumulative or metered value sample type as untotalled units don't have corresponding totals or rates.

If only part of a time slot has recorded data, the value for the slot should be the mean of data recorded, so the value used will be the average of the recorded data. For example, if the sample value is 30°C starting at 00:27 in a half-hour slot, then the value for that slot should be 30°C as that is the average of the recorded values.

Totalled

This is used for units such as power or rate of flow, where a total over time is meaningful and obtained by integrating. If multiple sample points occur in a time slot, the value is averaged over the time slot. The total value is the integral of the interpolated value over time; the derived values are adjusted so integrating them gives the same value as linear integration of the sample points.

If only part of a time slot has recorded data, the total for the slot should be the total of data recorded, so the value used will be the total of the recorded data divided by the length of the slot. For example, if the sample value is 30kW starting at 00:27 in a half-hour slot, then the total for that slot should be 30kW×3mins/60 = 1.5kWh so the value for the slot is 1.5kWh/(30/60h) = 3kW.

Independent

This is used for units such as cost, where a total over time is the simple sum of the values over the time. If multiple sample points occur in a time slot, the value for the time slot is the sum of the values in the slot. The total value is the sum of all sample values over time; the derived values are adjusted so that summing them gives the sum of the original sample points.

As far as downsampling goes, totalled and untotalled are treated the same - the difference between them appears when totalling.

Derived Spot Value, Untotalled

Untotatalled spot value values are based on a linear interpolation between the sample points.

If no sample points appear in a slot, no value is returned. These slots can be filled in by interpolation or an expression.

If there is a gap greater than the max gap period before or after a point, then its value is used to pad the value for the rest of the slot. This is shown in the first and final slots.

Values are the area under the spline divided by the width of the interval.

Derived data result ( Spot Value, Untotalled )

time	00:00	00:15	00:30	00:45	01:00	01:15	01:30	01:45
value	  8.3	  8.5     4.1       -       -       -       -     6.0

Derived Spot Value, Totalled

Totatalled spot value values are based on a linear interpolation between the sample points.

If no sample points appear in a slot, no value is returned. These slots can be filled in by interpolation or an expression.

If there is a gap greater than the max gap period before or after a point, then value of the slot is based on the proportion which is populated. This is shown in the first and final slots. The final slot has no value as the data stops and the total for the slot is not extrapolated.

Values are the area under the spline divided by the width of the interval.

Derived data result ( Spot Value, Totalled )

time	00:00	00:15	00:30	00:45	01:00	01:15	01:30	01:45
value	  7.5	  8.5     4.1       -       -       -       -       -

Derived Spot Value, Independent

Independent spot value values are based on summing the sample values within the time slots.

If no sample points appear in a slot, no value is returned. These slots can be filled in by an expression. Independent values are not normally interpolated - if you have receipts for £5 on Tuesday and £10 on Thursday it does not mean you spent £7.50 on Wednesday.

Values are the sum of the areas of the bars.

Derived data result ( Spot Value, Independent )

time	00:00	00:15	00:30	00:45	01:00	01:15	01:30	01:45
value	 16.0	 18.0     4.0       -       -       -       -     6.0

Derived Average, Totalled/Untotalled

Totatalled average values are based on a the average of samples values reported in the time slot. Time slots include their start time and exclude their end time.

If no sample points appear in a slot, no value is returned. These slots can be filled in by interpolation or an expression.

Usually you would use Average sample mode where you have data which is at the same or greater period as the building sample period, so processing for multiple values within a time slot is not the priority compared to preventing smoothing.

Derived data result ( Average, Totalled/Untotalled )

time			00:00	00:15	00:30	00:45	01:00	01:15	01:30	01:45
value			  8.0	  9.0     4.0       -       -       -       -     6.0
interpolated	  8.0	  9.0     4.0     4.5     4.9     5.3     5.8     6.0

Derived Average, Independent

Independent average values are based on summing the sample values within the time slots in the same way that Spot Value samples are treated.

If no sample points appear in a slot, no value is returned. These slots can be filled in by interpolation or an expression. Interpolation takes into account the timestamps of the sample points.

Values are the sum of the areas of the bars.

Derived data result ( Average, Independent )

time	00:00	00:15	00:30	00:45	01:00	01:15	01:30	01:45
value	 16.0	 18.0     4.0       -       -       -       -     6.0

Derived Cumulative, Totalled/Untotalled

Totatalled cumulative values are based on treating the the positive difference between sample values divided by their time difference in hours as the value of interest, and calculating the value based on averaging this out within the time slot. Time slots include their start time and exclude their end time.

If no sample points appear in a slot, no value is returned. These slots can be filled in by interpolation or an expression.

The orange line represents rate between values, in units per quarter hour. The contribution of the value is the gradient of the orange line. If the right hand point's value is less than the left hand point's value, then it is treated as a the counter having been reset to zero, so the positive difference is calculated as 4. The difference in time is 6 minutes. 4 units/6 minutes is 10 units/15 mins so the first part of the red area has a height of 10. The average height of the red areas in the first time slot is 12, so the average rate is 12/15 units per minute or 48 units per hour.

Derived data result ( Cumulative, Totalled/Untotalled )

time	00:00	00:15	00:30	00:45	01:00	01:15	01:30	01:45
value	 48.0	 24.0     0.6     1.1     1.1     1.1     1.1       -

Derived Cumulative, Independent

Independent values are based on summing the sample values within the time slots in the same way that Spot Value samples are treated.

If no sample points appear in a slot, no value is returned. These slots can be filled in by interpolation or an expression.

The bars are the differences between the start and end of the line segment they are placed at the end of.

The value for a time slot is the sum of the areas of the bars in the time slot.

Derived data result ( Cumulative, Independent )

time	00:00	00:15	00:30	00:45	01:00	01:15	01:30	01:45
value	  4.0	 14.0     0.0       -       -       -       -     2.0

Derived Metered Value, Totalled/Untotalled

Totalled metered values are based on treating the raw value divided by the time difference in hours since the previous value as the value of interest, and calculating the value based on averaging this out within the time slot. Time slots include their start time and exclude their end time.

The first metered value is ignored as it is not known at what time the period it covers starts. You can manually add a zero at the time the meter was started to cause the first real metered value to be used.

The slope of the orange line is the rate due to metered value divided by the interval between values.

The contribution of the value is the gradient of the orange line. For example, the first two points (1,12) and (7,4) are 6 minutes apart or 1/10 of an hour, so the slope is 4×10 or 40.

The average height of the red areas in the first time slot is 15.2, so the average rate is 15.2/15 units per minute or 60.8 units per hour.

Derived data result ( Metered Value, Totalled/Untotalled )

time	00:00	00:15	00:30	00:45	01:00	01:15	01:30	01:45
value	 60.8	 32.0    14.0     5.3     5.3     5.3     5.3       -

Interpolated data result ( Metered Value, Totalled/Untotalled )

time	00:00	00:15	00:30	00:45	01:00	01:15	01:30	01:45
value	 60.8	 32.0    14.0     5.3     5.3     5.3     5.3       -

Derived Metered Value, Independent

Independent metered values are based on summing the sample values within the time slots then converting to a regular value which would result in the same total since the last sample value.

This means if you have receipts for £5 on Tuesday and £10 on Thursday the values would be smoothed out so it reported an interpolated value of £5 for Tuesday, £5 for Wednesday, and £5 for Thursday

If no sample points appear in a slot, no value is returned. These slots can be filled in by an expression.

Gaps greater than five times the expected sample period of the channel will not be interpolated.

The values are the sum of the areas of the bars.

Derived data result ( Metered Value, Independent )

time	00:00	00:15	00:30	00:45	01:00	01:15	01:30	01:45
value	 16.0	 18.0     4.0       -       -       -       -     6.0

When interpolating, gaps are filled by spreading the value for the point back to the previous point.

The values are the sum of the areas of the orange blocks, which have the same area as the bars but spread across from the previous sample point. The values from the first point is put wholly in the slot in which it is recorded.

Interpolated data result ( Metered Value, Independent )

time	00:00	00:15	00:30	00:45	01:00	01:15	01:30	01:45
value	 27.2	 8.0      3.5    1.3     1.3     1.3     1.3     -

Derived Discrete stepping, Totalled

Discrete stepping values hold their values until a new sample point is encountered. Time slots include their start time and exclude their end time.

If data is present only for the latter half of a slot, then the value is the average for the whole of the slot

• if the value is 30kW starting at 00:27 in a half-hour slot, then the total for that slot should be 30kW×3mins/60 = 1.5kWh so the value for the slot is 1.5kWh/(30/60h) = 3kW.

The orange line represents the sample points interpolated by holding the value until the next point is encountered.

The derived value is the average value this line within the time slot, or the red area divided by the width of the interval.

Derived data result ( Discrete stepping, Totalled/Untotalled )

time	00:00	00:15	00:30	00:45	01:00	01:15	01:30	01:45
value	  6.9	 10.7     4.0     4.0     4.0     4.0     4.0     6.0

Derived Discrete stepping, Untotalled

Discrete stepping values hold their values until a new sample point is encountered. Time slots include their start time and exclude their end time.

If data is present only for the latter half of a slot, then the value is the average for the proportion of the slot which has data. This prevents the first value in the sample series being reduced

• if the value is 30°C starting at 00:27 in a half-hour slot, then the value for that slot should be 30°C.

The orange line represents the sample points interpolated by holding the value until the next point is encountered.

The derived value is the average value this line within the time slot, or the red area divided by the width of the interval. For example, for the first slot the value is (12×6 + 4×8)/(6+8) = 7.4.

Derived data result ( Discrete stepping, Totalled/Untotalled )

time	00:00	00:15	00:30	00:45	01:00	01:15	01:30	01:45
value	  7.4	 10.7     4.0     4.0     4.0     4.0     4.0     6.0
	

Derived Discrete stepping, Independent

It is probably not meaningful to use discrete stepping with independent values, as discrete stepping implies the value is held for a period of time rather than being time independent.

The behaviour is the same for the untotalled mode.