Streamlining Backend Service Costs: A Guide for Cost Reduction

Picture this: you’ve just wrapped up another sprint planning meeting, feeling a sense of accomplishment and readiness to tackle the tasks ahead. But then, your manager approaches you with a new high-priority card that reads, “Reduce the cost of running the service.” A wave of uncertainty washes over you as you scratch your head, wondering where to begin.

This scenario has played out many times in my career as a software engineer. Through trial and error and from some of the best engineers, I’ve learned a set of strategies that have consistently proven effective in reducing cloud expenses and achieving significant cost savings.

In this blog, I’ll share my insights and practical strategies to help you navigate the terrain of backend service cost optimization.

Let’s begin.

Identify all the components in your backend

To effectively optimize costs, it’s essential to comprehensively identify all the components involved in your backend infrastructure. Each of these components contributes to the overall cost of running your backend services. The common backend components that are cost centers are listed in the image below. We will further explore specific cost-saving strategies for each of these key components.

Measuring the cost

What gets measured, gets managed

The next step towards cost reduction is measuring the current cost of running your backend service. While this sounds straightforward, it is often difficult to measure what really is the cost of all the components involved while running a service. Different components of a service could be on different cloud platforms and very likely there are also a bunch of SAAS applications for error tracking, logs, on-call management, etc.

To accurately track and monitor costs, consider utilizing cloud-based dashboards or dedicated cost management tools. Major cloud service providers such as AWS, GCP, and Azure offer their own cost dashboards along with APIs for retrieving cost data. You can also create a customized dashboard that suits your company’s requirements, giving you a clear view of your service costs in one place.

Having the right data in a consumable format empowers engineers to quickly figure out what are the highest cost centers and provides a lot of direction and actionable. Spotify built a really interesting tool for the same.

Implement cost-saving tactics, focusing on one component at a time.

Once you have a view of the cost of running every component of the backend service, identify the ones that make the most significant contributions. These components often present the most potential for enhancements and cost reduction

There are certain strategies and things you can look out for in each component in order to optimize costs. Let’s take every component and dive deeper.

Services

Services are the core components of the backend infrastructure, responsible for handling incoming requests and fulfilling business logic. Optimizing services is crucial for reducing costs and improving performance. Here are some key strategies to consider:

Services
|
|----Right-Sizing for Resource Efficiency
|----Auto scaling
|----Consider different configurations for business and non-business hours
|----Consider splitting the service based on access pattern
|        |----Splitting a Read-Write Service
|        |----Splitting a High-Traffic/Low-Traffic components
|        |----Splitting a Service Based on User Roles
|----Leverage spot instances
|----Switch off idle services

Right-sizing for resource efficiency

Right-sizing involves ensuring that your service instances are provisioned with the appropriate amount of compute resources (CPU, memory) to meet demand without wasting resources. Have the right dashboards to monitor the usage of CPU and memory. Look for any case of overprovision and reduce them.

Auto Scaling

Having an autoscaling solution that will dynamically adjust the number of services to be run based on the traffic patterns/CPU and memory consumed or on some custom metrics is a must. This will ensure that at all times, only the required number of services run and costs accordingly. Most common solutions come with their own autoscaling capability like Kube HPA, Amazon EC2 Auto scaling, etc.

Consider different configurations for business and non-business hours

Many businesses experience different traffic patterns during business and non-business hours. To optimize costs, consider using different service configurations during these periods.

Consider splitting the service based on the access pattern

Splitting a service based on access pattern is a technique for optimizing resource utilization and performance by dividing a service into smaller, independent services that cater to different access patterns. The main key is to identify on what basis the service should be split.

Examples of splitting services by access pattern

• Splitting a read-write service: Divide a service into separate read-and-write components to optimize resource allocation based on access patterns.
• Splitting high-traffic/low-traffic components: Separate high-traffic and low-traffic components to ensure that resources are allocated effectively. High-traffic components can be scaled independently to handle demand, while low-traffic components can be provisioned with fewer resources to reduce costs.
• Splitting a service based on user roles: Divide the service into components based on user roles to provide differentiated access and performance. This can be particularly useful for services that cater to users with different access privileges or usage patterns.

Leverage spot instances

Consider using spot instances for non-critical workloads like staging to further reduce costs. Spot instances are spare compute capacity in the cloud that is available at a significantly lower price than on-demand instances.

Switch off idle services

Terminate the instances when they are not in use. In one of the companies I worked for, the staging would be brought down every night for 10 hours as no one generally works during these hours. This is an easy 40% cost reduction if it will work for your company.

Databases

Most backend applications require a data storage solution. The specific optimization strategies for databases depend on the database type itself. However, some general principles can be applied to optimize any database for cost efficiency.

Databases
|
|---- Right-sizing
|        |---- Provisioned DB Size
|        |---- Tier/Plan
|        |---- Scaling based on demand
|---- Query Optimization
|        |---- Identify long-running queries
|        |---- Optimize indexes
|        |---- Identify cost-contributing tables and queries
|---- Data Archiving
|---- Data Partitioning

Right-sizing

Provisioned DB size: Evaluate the provisioned DB size and adjust it to match actual usage patterns. Estimate the amount of data you will store and the performance required. Avoid over-provisioning, as this can lead to unnecessary costs.

Choose the right tier/plan: Choose the appropriate tier or plan for your database based on your performance and storage requirements. Consider factors such as IOPS (Input/Output Operations Per Second) and storage capacity when selecting a tier or plan.

Scaling instances up or down based on demand: Implement auto-scaling rules to automatically scale database instances up or down based on real-time usage patterns. This ensures that instances are always sized appropriately to meet demand.

Query Optimization

Improve long-running queries: Identify long-running queries that consume significant resources. Optimize these queries to improve performance and reduce resource consumption.

Index optimization: Assess the need for indexes and evaluate their effectiveness. Remove unnecessary indexes, as they can impact performance and increase storage costs.

Optimize cost-contributing tables and queries: Identify tables and queries that contribute the most to database costs. Evaluate their necessity and consider alternative approaches to reduce their cost impact. This may involve data partitioning, data archiving, or using more cost-effective storage options.

Data archiving: Archive infrequently accessed data to lower-cost storage tiers, such as cold storage or archival storage. This can significantly reduce storage costs without impacting performance for frequently accessed data.

Data partitioning: Partitioning the data into smaller, more manageable segments, allowing for more efficient data access can reduce the overall workload on the databases and sometimes reduce the cost as well.

Caches

Caches are frequently employed in the backend to minimize latency and reduce the burden on the primary database.

Caches
|
|-----Right-sizing
|        |----Determine appropriate cache size
|        |----Choose appropriate tier or plan
|-----Avoide storing large items
|        |----Define item size threshold
|-----Expiration policies
|        |----Implement TTL-based expiration
|-----Sharing cache instances
|        |----Service consolidation
|        |----Cache partitioning
|-----Hot Key Management
|        |----Identify hot keys
|        |----Implement cache eviction policies

Right-sizing: Determine the appropriate cache size based on the frequency and volume of data access.

Choose the right tier/plan: Choose the appropriate tier or plan for your cache based on your performance and latency requirements. Consider factors such as cache hit rates and access latency when selecting a tier or plan.

Avoid storing large items

Define a size threshold for cacheable data items to avoid storing excessively large items in the cache. Avoid storing entire objects in the cache and only store the required information. Having large items in the cache causes performance degradation and unnecessary resource consumption.

Expiration policies

Implement expiration policies to automatically remove data from the cache based on time-to-live (TTL) values.

Sharing cache instances

• Service consolidation: Evaluate whether multiple services can share a single cache instance to reduce redundancy and cost. This can be particularly beneficial for services that have similar data access patterns.
• Cache partitioning: Consider partitioning large caches to improve performance and prevent cache thrashing.

Hotkey management

• Hot Key Identification: Identify hotkeys, which are frequently accessed data items, and consider alternative storage mechanisms for them. This may involve storing hotkeys in a separate, high-performance cache or using a distributed cache solution.
• Consider implementing policies to evict hotkeys.

Logs

While discussing the topic of logs, I’m reminded of an incident involving a service I once worked on. It turned out that almost two-thirds of the logs generated by this service were merely “test” logs that someone had created during the initial development phase. These test logs had never been removed, and it was quite surprising when our team received an alert indicating that we were one of the primary contributors to the overall logging costs.

Logs
|
|-----Reducing Unwanted Log Generation
|        |----Remove unwanted logs
|        |----Configure right log level
|        |----Use log filtering
|        |----Use conditional logging
|-----Optimise debug mode 
|        |----Disable debug mode by default
|        |----Define guidelines for debug logs
|-----Log archiving
|        |----Compress and archive logs
|        |----Delete logs periodically

Reducing unwanted log generation

• Remove unwanted logs :)
• Log level configuration: Carefully configure log levels to generate only the necessary logs. Avoid generating excessive logs at higher levels (INFO, DEBUG) that can increase storage and processing costs.
• Log filtering: Implement log filtering mechanisms to exclude irrelevant or unnecessary log entries.
• Conditional logging: Employ conditional logging to generate logs only when specific conditions are met.

Optimizing debug mode

• Disable debug mode by default
• Define clear guidelines and restrictions for enabling debug mode to prevent its overuse and excessive log generation.

Log archiving

Regularly compress log files to reduce storage space and associated costs. Move compressed log files to low-cost storage solutions. Delete logs that are no longer needed periodically.

Metrics

Who watches the watchman

Metrics
|
|-----Monitor metrics size
|        |----Cardinality Analysis
|        |----Metrics monitoring 
|-----Assess Metric Necessity
|        |----Periodically assess the need of metrics
|-----Metrics retention policy
|-----Cost allocation per service/team

Monitor metrics size

• Cardinality analysis: Regularly analyze metrics to identify high cardinality metrics, which are metrics with a large number of unique values. These metrics can consume significant storage space and contribute to increased costs.
• Metric monitoring: Continuously monitor metric storage usage to identify trends and detect spikes in storage consumption. This can help proactively address potential issues related to high cardinality metrics.

Assess metrics necessity

Periodically review all metrics to assess their necessity and determine if they are still providing valuable insights. Eliminate or reduce the collection of metrics that are no longer required or are not providing sufficient value.

Metric retention policies

Establish clear metric retention policies to define how long different types of metrics are stored. Delete or archive older metrics that are no longer required to reduce storage costs.

Cost allocation

Consider implementing a cost allocation mechanism to charge different teams or applications for the storage costs associated with their metrics. This can incentivise more efficient metric usage and reduce overall storage expenses.

Vendors

Regularly review software licenses and subscriptions to ensure you’re utilizing the most cost-effective options. Consider open-source alternatives or negotiate better licensing agreements with vendors. Try to standardize the software used across teams and centralize procurement.

CI/CD

CI/CD
|
|-----Leverage shared runnings
|-----Right size resources
|-----Use parallelisation
|-----Spot instances for CI/CD
|-----Optimise tests

• Leverage shared runners to reduce the need for dedicated runners for each project or team. Shared runners can be used for multiple projects, improving resource utilization and reducing costs.
• Right size the resources and enable auto-scaling
• Use parallelization effectively to speed up builds and reduce the time CI/CD resources are active. This can lead to significant cost savings, especially in cloud-based CI/CD services.
• Consider spinning up CI/CD resources using spot instances. This could sometimes increase the time it takes for your build to run, but worth it sometimes.
• Optimise tests — Review and optimize your test suits to reduce the time required to run them. Remove all unwanted and redundant tests.

Code

Unlocking savings is sometimes just a code away.

Sometimes the major cost incurred is not because of any specific component, but the code itself. It could all be because the code we have written is not performant enough. Identifying the performance bottlenecks in the app and fixing those could result in tangible cost benefits.

Continuously monitor and alert on cost

Configure cost alerts to receive notifications when your cloud costs exceed predefined thresholds and define an SOP on how these would be handled.

Cultivate a culture of cost optimization

Cost optimization is not a one-time activity but a mindset shift. Companies can immensely benefit when a culture of cost consciousness is built. It also helps engineers make more informed decisions.

I’ve been in workplaces where we made a strong effort to encourage a culture of saving money, and here are some strategies that I felt really worked

• Share the actual expenditure figures openly, allowing everyone to understand the financial realities.
• Provide engineers with the right set of tools to understand the cost trends, and some actionable wherever possible.
• Periodically review the expenses of running your service, like every three weeks.
• Define clear and realistic budgets at a quarter level.
• Encourage engineers to consider costs while proposing new solutions.
• Celebrate when costs are reduced.
• Recognize that optimizing costs is often a challenge. Give engineers the time and support they need to navigate and solve these complexities effectively.

For further reading, I recommend exploring an article on this topic authored by Amazon here.

Conclusion

In today’s business landscape, reducing costs across all departments is a crucial objective for many organizations seeking to maintain profitability. The backend represents a significant opportunity for cost reduction.

Like everything in engineering, cost reduction strategies also come with their own tradeoffs. There is no one-size-fits-all solution and we need to carefully evaluate before changing anything in our production systems. The strategies outlined in this guide are aimed to provide a starting point for reducing backend service costs.

Cost optimization is not a one-time project but an ongoing process that requires continuous monitoring, evaluation, and adaptation.

I would like to thank Udaan and Freshworks for giving me the opportunity to learn about these techniques.

These are some additional blogs on this topic —

Zoom, Spotify and others slashed their cloud costs by millions - how did they do it?

10 things you can do today to reduce AWS costs | Amazon Web Services

Saving Millions on Logging: Finding Relevant Savings

Data Engineering Best Practices: How Netflix Keeps Its Data Infrastructure Cost-Effective