Blog

Moonshadow Reduces Data Carbon Footprint by 90%

By |July 3rd, 2024|

Climate change is recognized globally as a threat to humanity. Almost all industries are working hard to innovate us out of the heat trap. Engineering changes are reducing our carbon footprint from driving vehicles to pouring concrete. There is one area, however, where innovation is leading to a steep increase in carbon emissions: data.

Image and Data courtesy of McKinsey Company

According to McKinsey US data center power consumption increased from six gigawatts in 2014 to 17 gigawatts in 2022. McKinsey projects that it will double again by 2030. Whereas data centers used 3% of power in the US in 2022 this […]

Read More

MAG is the First Public Agency to Implement MMZIP

By |February 13th, 2024|

One week of connected vehicle data for Maricopa contains 100 million records

The Maricopa Association of Governments (MAG) is a Council of Governments (COG) that serves as the regional planning agency for the metropolitan Phoenix area. MAG utilizes connected vehicle data for transportation planning and transportation system analysis purposes. The data consists of millions of anonymized trips on regional roads. The data include large, daily files and come to more than 25 Terabytes in size for a few years period. Managing and using this data is a challenge. One essential function that needs to be done before the data can be used […]

Read More

How to Compress Time to (Almost) Nothing

By |November 13th, 2023|

When working with time-series data size matters. If we can reduce the size of a database by 90% storage and transfer costs go down by 90%. What’s more, the data can be transferred or loaded 10 times faster.

Moonshadow Mobile introduced MMZIP earlier this year and we started reporting compression ratios that are much higher than those of existing technologies such as GZIP or BZIP2. Some people have asked us how this is possible. In this post we’ll share some insights on how we achieve these high compression ratios. All time-series data includes a field to store the date and time […]

Read More

MMZIP Filters Two Billion Waypoints per Hour

By |September 20th, 2023|

When using connected vehicle data for transportation analytics one of the biggest challenges is the sheer size of the data. A typical dataset with a month of data can contain five billion waypoint records and working with datasets this size is difficult and time consuming. The good news is that any individual transportation analytics project dataset only uses a small percentage of the data. For analyzing the freeway use in a county, for instance, we want to create a dataset that only contains the waypoints in the county and only those that are generated while the vehicle was driving on […]

Read More

MMZIP Enables CCPA & GDPR Compliance

By |March 14th, 2023|

Both the California Consumer Privacy Act (CCPA) and Europe’s General Data Protection Regulation (GDPR) include the “right to be forgotten.” Consumers in California and Europe can ask organizations to delete all data about them and businesses are required to comply with each request. This is challenging enough for databases built around people where the primary key is a personID and all data about a person is linked to that. It can be difficult to locate all backups and remove all data to a person from those. Many databases are updated automatically from external sources and organizations need to put filters in […]

Read More

MMZIP: Streaming Waypoint Compression

By |February 14th, 2023|

Or how to compress terabytes of data in 32GB of RAM

A waypoint dataset for an area can quickly exceed ten billion records. An uncompressed dataset with ten billion waypoints will take three terabytes of space or more depending on the data in the columns. Servers with terabytes of RAM are expensive; Amazon AWS charges $13 per hour for a 2TB server which comes to over $9,000 per month. Servers with 32GB of RAM start around $0.35 per hour, or $250 per month, so there is a big cost incentive to compress and decompress large datasets on servers with little memory.

Moonshadow’s […]

Read More