Building Technology at OLX India

From Events to Insights — EventHouse Does It Best

Ever wondered how apps evolve to create effortless user experiences?

Let’s dive in.

Imagine this: You’re a seller trying to list your product. You take photos, pick a category, choose a location, and finalize your post.

Sounds simple, right? But what if something along the way frustrates you? You hesitate, and you drop off without ever listing your item. Now, let’s say our system only tracks the first and last steps. We know some users abandon the process, but we have no clue where or why. Are they struggling with the image upload? Is category selection confusing? Without the right data, we’re left guessing instead of improving. But what if we collect insights at every step? Suddenly, patterns emerge. We pinpoint the exact friction points, smooth them out, and watch conversion rates soar.

The graphs below reveal user drop-offs at each step of the ad posting process. By analyzing these trends, we can identify friction points and improve the seller experience for higher conversions.

1. Day to Day: Seller Drop-Off Trends: A sharp decline on Day 2 highlights potential issues affecting ad completion rates.
2. Variant A vs Variant B: A/B Testing Insights: Comparing two versions of the seller flow to determine the better-performing user journey.

Let’s understand events and their tracking to make further discussions more meaningful.

An event refers to any user action or system-generated activity that occurs within a digital application, website, or platform. Events capture interactions like clicks, scrolls, form submissions, video plays, purchases, or even backend processes such as API calls or data updates.

Event tracking is the process of capturing, storing, and analyzing these events to understand user behaviour, optimize experiences, and improve business outcomes. This is done using analytics tools, tracking pixels, or event-driven architectures.

Why Event Tracking Matters

Every action a user takes on a digital platform, clicking a button, filling out a form, or abandoning a cart is an event. Event tracking helps businesses understand user behaviour, optimize funnels, and make data-driven decisions. But what happens when your event-tracking system has limitations? You miss critical insights, and optimization becomes guesswork rather than a strategic approach.

The Challenge with Existing Event Tracking Systems

Many businesses rely on third-party event-tracking tools to monitor user interactions. These platforms offer pre-built solutions but often come with restrictions on the number of events, data storage constraints, and high costs.
At OLX India, we serve over 3M+ daily active users (DAUs) and 30M+ monthly active users (MAUs). Our existing event tracking system allowed us to track only ~2.8 million events per day far too low to capture the complete user journey. This restriction meant that critical user interactions went untracked, making it difficult to optimize funnels, reduce drop-offs, and make informed business decisions.

This is where EventHouse comes in.

Rather than relying on costly third-party tools, our engineering team built EventHouse a scalable, efficient, and cost-effective event-tracking platform tailored to our needs.

With EventHouse, we:

• Capture the entire user journey without restrictions.
• Gain deeper insights into user behaviour for data-driven decisions.
• Optimize processes by identifying drop-off points in real-time.
• Reduce reliance on external tools, saving costs while increasing efficiency.

The shift to EventHouse wasn’t just about gaining more insights it was about making event tracking sustainable. We aimed to increase event tracking by 10x while reducing costs significantly, ensuring complete visibility without financial strain.

By optimizing every stage, from event collection to processing and storage, EventHouse now provides limitless event tracking, empowering product and marketing teams to refine user experiences confidently.

The Future of Event Tracking

Event tracking isn’t just about collecting data, it’s about unlocking opportunities. With EventHouse, we’ve taken control of our event data, ensuring we have the insights needed to enhance user experience, improve conversions, and drive business growth.

For any business looking to optimize its platform, event tracking is not optional, it’s essential. And having the right system in place can be a game-changer. This transformation is helping OLX India unlock unprecedented levels of analytics while ensuring scalability, efficiency, and financial sustainability.

How is EventHouse Improving Our System?

1. Enhanced User Journey Analysis: By collecting comprehensive event data, we can analyze user funnels more deeply, identifying pain points and areas for improvement. Unlike Google Analytics (GA) and Clevertap, which impose limitations on using event parameters, EventHouse allows us to track and analyze events with rich contextual data, providing a clearer picture of user behaviour.
2. Deeper Data-Driven Decision Making: One of GA’s key drawbacks is the inability to use raw event data for analysis, limiting deeper insights. With EventHouse, we can directly access raw event data, enabling us to join it with transactional data in our database to uncover meaningful trends and patterns. This unlocks advanced analytics capabilities that were previously restricted.
3. Improved Revenue and Conversions: Understanding user behaviour at a granular level allows us to refine strategies that drive revenue growth, improve user engagement, and boost paid user conversions. Since EventHouse provides unsampled data, we ensure accurate insights, something not always possible with external analytics platforms.
4. Automated and Custom Reports: EventHouse allows us to build automated reports, eliminating manual effort and making analytics more efficient. With real-time tracking and dashboarding capabilities, teams can instantly access the insights they need, without the delays and constraints of third-party tools.
5. Scalability and Cost Efficiency: Unlike Clevertap, which becomes increasingly expensive as event tracking scales, EventHouse allows us to track unlimited events at a fraction of the cost. This ensures we can expand our analytics capabilities without financial constraints, making it a more sustainable and budget-friendly solution.
6. The Next Holy Grail of Analytics: EventHouse is not just another analytics tool—it has the potential to become our single source of truth for accurate, unsampled event data. By overcoming the limitations of GA and Clevertap, we now have a powerful system that ensures scalability, flexibility, and deeper insights, giving us a competitive edge.

With EventHouse in place, we will have a powerful tool to collect and leverage data for improving our product, enhancing user satisfaction, and driving business growth.

How Are We Collecting Data In-House?

The EventHouse SDK provides a centralized and efficient way to collect, process, and send analytics events across Android, iOS, and Web platforms. It standardizes event collection, helping the company make data-driven decisions.

EventHouseViewModel –Data Fetching and Caching Strategy

• Fetches event configurations from an external API. If we need to update event tracking rules, we can modify them at the API level without requiring app updates. Ensures all clients always fetch the latest event configurations automatically.
• Caches data in memory to provide instant access to event configurations without hitting the API repeatedly. and persists it in a local database, In case the app restarts or the cache is cleared, the event configurations remain available.
• Uses ETag-based change detection to minimize unnecessary API calls. Instead of downloading the full data set every time, we send the ETag in the If-None-Match header of subsequent requests. So that we fetch only the updated content, keeping our system up-to-date with minimal bandwidth usage.

EventScheduler

• Manages event processing using a buffered queue. Directly sending every event to the server would cause excessive network requests, increasing latency and server load.
• Sends events when the queue reaches a predefined size or after 20 seconds. Without a rate limit, frequent event bursts could overload the server and degrade performance
• Ensures thread safety with a Mutex and runs asynchronously via a coroutine scope.  Without a Mutex, multiple threads could modify the queue at the same time, causing data inconsistencies or crashes.
• Validates events before queuing and allows manual flushing of all stored events.

EventQueue

• Temporarily stores events in an ArrayDeque. An ArrayDeque allows fast insertion and removal from both ends, making it more efficient than traditional lists or arrays.
• Monitors queue overflow and rate limits.
• Sends events when buffer size exceeds or time threshold is met, then clears the queue.

SessionManager

• Manages session creation and lifecycle. A combination of UUID (universally unique identifier) and timestamp ensures that each session ID is unique, preventing collisions and maintaining session integrity.
• Creates new sessions after 30 seconds of inactivity. A short timeout window ensures that each session represents an active user interaction while preventing unnecessary session fragmentation.
• Ensures session validity and allows manual session resets when needed. Some scenarios (e.g., user logout, account switch) require immediate session termination rather than waiting for inactivity.

API Design

Designing a scalable and efficient event-tracking system requires careful handling of initialization, batching, and data flushing. The EventHouse API is built to ensure seamless event processing, optimized network usage, and data integrity while minimizing system overhead.
Events are sent when the queue reaches its bufferSize. Sending every event individually would lead to excessive API calls, causing network congestion and slowing performance. The flushAll() function also ensures that any remaining events are sent when needed, preventing data loss. Automatic flushing prevents delays in event processing, while manual flushing gives developers control over event transmission in critical scenarios.

Collecting Events from Apps

We have developed an innovative event collection & ingestion system that is highly cost-efficient to run and efficiently collects, processes, and stores event data in Amazon S3 for downstream Data Engineering (DE) pipelines.

1. Client SDK: Smart Event Collection and Compression

The first step in our pipeline is the event collection process. We’ve developed a dynamic Client SDK that can be configured to collect a flexible number of events from client applications. What sets this SDK apart is its ability to send bulk events in a compressed format. By using compression, we’ve significantly reduced the amount of ingress traffic and saved 90% in costs related to Application Load Balancer (ALB) usage. This approach ensures that we can handle massive event streams without overwhelming the network or incurring steep costs.

2. Application Load Balancer (ALB): Efficient Traffic Management and Secure Routing

Once events are sent from the client, they pass through the Application Load Balancer (ALB). The ALB plays a critical role in managing the incoming traffic. It balances the load by distributing requests across backend services, ensuring optimal performance and preventing any one server from becoming a bottleneck. Additionally, the ALB acts as an API Gateway, securely routing requests to appropriate backend services. Without the ALB, direct transmission of logs to backend servers could lead to performance bottlenecks, making the infrastructure inefficient.

3. Nginx Server: Event Processor & Aggregator

After the ALB, the logs are forwarded to Nginx, which acts as a lightweight reverse proxy and log collector. Nginx is deployed as Kubernetes pods to manage high availability and scalability. It runs scripts to collect event logs and store them in event files. These files are rotated based on size to ensure efficient log management. By utilizing Nginx in this way, we can efficiently handle incoming requests, process them quickly, and log them for future analysis without overloading the system.

Cost Efficiency & Scalability of Nginx Servers – One Nginx Server can handle up to 1M RPM, with added event processing logic our servers are capable of handling more than 300K RPM with a single CPU and 1G of Memory. Which makes this system design highly cost-efficient.

4. FluentBit: Event Distributor

Next in the pipeline is FluentBit, a powerful and lightweight data processor & distributor. FluentBit picks event data files from Nginx Servers, compresses them and sends them to Amazon S3 at regularly defined intervals and defined file path format. By aggregating events and processing them before storage, we’re able to make data storage more efficient and improve overall ingestion performance down the line.

5. AWS S3: Scalable and Reliable Event Storage

For storing event data, we leverage AWS S3, a highly scalable and reliable object storage service. The events are stored in compressed JSON Lines format (.json.gz), which offers a balance of human readability and storage efficiency. While formats like Parquet offer better query performance and compression, JSON lines remain our choice for easy debugging and log inspection. The compression helps reduce storage costs while keeping the data accessible and efficient for processing.

• Compression and Storage Efficiency: Each event is individually compressed using the GZIP (.gz) format, which allows us to reduce storage space while keeping the data easily accessible. This compression approach not only minimizes storage costs but also accelerates data transfer when accessing logs or moving them through different stages of the pipeline.
• Data Partitioning: Optimizing for Future Growth: We partition the stored event data by date (Year/Month/Date/Hour). This partitioning strategy brings several benefits:
• Simplified Data Management: It becomes easier to retire historical data or manage storage tiers over time.
• Enhanced Query Performance: If we choose to adopt services like Amazon Athena for querying, this partitioning structure will already be optimized for high performance.
• Faster Debugging: By organizing data in this way, debugging intermittent issues becomes simpler, as we can quickly reload or inspect data from specific periods.

Why Does This Architecture Make Sense?

This architecture is designed to be both scalable and highly cost-efficient. Each component is optimized to handle increasing volumes of data while minimizing operational costs. By compressing event data early in the pipeline, using load balancing for traffic management, and structuring data storage intelligently, we’ve built a system that can grow with our needs without breaking the bank.

Compression at Every Step: From client SDK to storage, data is compressed, reducing storage costs and speeding up data transfer.

Scalable Infrastructure: With Kubernetes, Nginx, and AWS services, the system can easily scale to accommodate future growth in data volume and complexity.

Low-Cost Server: One Nginx Server is capable of processing over 500K RPM, providing a highly optimized and cost-efficient solution for request processing for event tracking systems.

Real-Time Processing: FluentBit allows us to preprocess data in real-time, improving ingestion speed and storage efficiency.

Data Engineer Pipeline to bring event data to Redshift

At OLX India, handling millions of events daily requires an intelligent and efficient data pipeline. It all begins with event and log files arriving in S3, a reliable and cost-effective storage solution designed for scalability. As soon as a new file lands, AWS Lambda springs into action, detecting the update in real time and setting the pipeline in motion without any manual intervention.

Instead of moving large files through the system, Lambda extracts only the metadata and streams it into Kinesis. This ensures that processing remains lightweight, fast, and highly efficient, even when dealing with high-velocity data. The actual files remain securely stored in S3, ready to be accessed when needed.

From there, a Python ETL pipeline continuously listens to Kinesis, fetching metadata and using it to retrieve the raw files from S3. These files are then seamlessly loaded into a Redshift staging table, allowing for structured validation and transformation before they reach the final production tables. This extra step ensures data integrity, preventing inconsistencies and optimizing performance for analytics and reporting.

To maintain seamless operations, a robust checkpointing system tracks the last processed file, preventing duplication and enabling easy recovery in case of failures. At the heart of this operation, Jenkins orchestrates the entire workflow, automating scheduling, execution, and monitoring while sending real-time alerts.

This worked so well for us that the staging tables we created were actually used as a view directly into s3 files for testing the entire data flow. The eventhouse tables in redshift follow a monthly partitioned naming convention, where the suffix represents the year and month. This helps us get rid of data that is too old without having to run compute heavy vacuum jobs on a single table over and again.

KEY TAKEAWAYS:

1. Unlimited Event Tracking Without Cost Barriers

• EventHouse enables unrestricted event tracking at One-Tenth the cost.
• Clevertap’s high costs restricted event tracking to 25 events with limited parameters. Scaling beyond this required significant additional expenses.

2. Optimized Data Processing & Storage

• Performant & Cost Effective Event Collector Design.
• AWS S3 with structured partitioning ensures scalability & cost efficiency.

3. High-Performance Event Processing

• Batching & buffered queues prevent network overload.
• Asynchronous scheduling ensures high throughput without lag.

4. Automated & Scalable Data Pipelines

• S3 triggers, Lambda, and Kinesis enable real-time ingestion.
• Python ETL & Redshift COPY command streamline transformation & storage.

5. Cost Efficiency Without Performance Trade-Offs

• Gzip compression & partitioned storage reduce costs while keeping data accessible.
• Autoscaling infrastructure optimizes resources based on demand.

6. Leverage Open-Source for Faster Development

• Used Fluentd, Vector, and Lua scripting to minimize build effort.
• Ensured scalability & reliability with proven open-source tools.

By applying these key strategies, EventHouse has empowered OLX India with a scalable, efficient, and cost-effective event-tracking system that enhances user insights, improves conversions, and drives business growth.

Acknowledgements

Development Contributions –

• Data Engineering (DE): Navdeep Gaur (Engineering Lead), Shrishty Srivastava (Engineer)
• Backend: Parvez Hassan (Backend Architect), Akash (Engineer)
• Android/iOS: Ravi Rawal (Chapter Lead), Sandeep Chhabra (Chapter Lead)
• PWA: Anshul Bansal (Chapter Lead), Yash Chitlangia (Engineer)
• DevOps: Kishan Tiwari (Lead), Pushpender Kumar (Architect)
• Analytics: Jatin Pal Raina
• QA: Rathod Singh, Gaurav Jolly
• Engineering Lead (DE): Hitesh Kumar
• Engineering Lead (Platform): Hitesh Das