Uncategorized

Moonshadow Reduces Data Carbon Footprint by 90%

By |July 3rd, 2024|

Climate change is recognized globally as a threat to humanity. Almost all industries are working hard to innovate us out of the heat trap. Engineering changes are reducing our carbon footprint from driving vehicles to pouring concrete. There is one area, however, where innovation is leading to a steep increase in carbon emissions: data.

Image and Data courtesy of McKinsey Company

According to McKinsey US data center power consumption increased from six gigawatts in 2014 to 17 gigawatts in 2022. McKinsey projects that it will double again by 2030. Whereas data centers used 3% of power in the US in 2022 this […]

Read More

MAG is the First Public Agency to Implement MMZIP

By |February 13th, 2024|

One week of connected vehicle data for Maricopa contains 100 million records

The Maricopa Association of Governments (MAG) is a Council of Governments (COG) that serves as the regional planning agency for the metropolitan Phoenix area. MAG utilizes connected vehicle data for transportation planning and transportation system analysis purposes. The data consists of millions of anonymized trips on regional roads. The data include large, daily files and come to more than 25 Terabytes in size for a few years period. Managing and using this data is a challenge. One essential function that needs to be done before the data can be used […]

Read More

How to Compress Time to (Almost) Nothing

By |November 13th, 2023|

When working with time-series data size matters. If we can reduce the size of a database by 90% storage and transfer costs go down by 90%. What’s more, the data can be transferred or loaded 10 times faster.

Moonshadow Mobile introduced MMZIP earlier this year and we started reporting compression ratios that are much higher than those of existing technologies such as GZIP or BZIP2. Some people have asked us how this is possible. In this post we’ll share some insights on how we achieve these high compression ratios. All time-series data includes a field to store the date and time […]

Read More

MMZIP Enables CCPA & GDPR Compliance

By |March 14th, 2023|

Both the California Consumer Privacy Act (CCPA) and Europe’s General Data Protection Regulation (GDPR) include the “right to be forgotten.” Consumers in California and Europe can ask organizations to delete all data about them and businesses are required to comply with each request. This is challenging enough for databases built around people where the primary key is a personID and all data about a person is linked to that. It can be difficult to locate all backups and remove all data to a person from those. Many databases are updated automatically from external sources and organizations need to put filters in […]

Read More

MMZIP: Streaming Waypoint Compression

By |February 14th, 2023|

Or how to compress terabytes of data in 32GB of RAM

A waypoint dataset for an area can quickly exceed ten billion records. An uncompressed dataset with ten billion waypoints will take three terabytes of space or more depending on the data in the columns. Servers with terabytes of RAM are expensive; Amazon AWS charges $13 per hour for a 2TB server which comes to over $9,000 per month. Servers with 32GB of RAM start around $0.35 per hour, or $250 per month, so there is a big cost incentive to compress and decompress large datasets on servers with little memory.

Moonshadow’s […]

Read More

MMZIP Reduces Wejo Data Preparation Time & Costs by 90%

By |December 14th, 2022|

December 14, 2022

Moonshadow Mobile started the development of MMZIP, its waypoint compression technology, last summer. We recently had an opportunity to test out MMZIP for a large-scale project that was using Wejo data. The results were very impressive; MMZIP reduces the data preparation time and costs by over 90% and our client can start using the data weeks earlier.

Moonshadow was asked to prepare one year of Wejo data for all trips in the state of Alabama for transportation analysis. Wejo generated over 90 billion waypoints and made these available as 10TB of GZIP files on AWS. Downloading the 10TB of […]

Read More