As a result, we could use lesser nodes and move our storage to Amazon Redshift managed storage. The most obvious reason for this was the decoupled storage from compute. The solutionĪfter discussions with AWS experts and the Amazon Redshift product team, we at Dream11 were recommended the most viable option of migrating our Amazon Redshift clusters from dc2.8xlarge to the newer RA3 nodes. Additionally, we were expecting more data volume over the next few years. Because we anticipated significant growth during the Indian Premier League (IPL) 2021, we actively explored various options using our AWS Enterprise Support team. Not only was this increasing our costs, it also meant that we were adding additional compute power when what we really needed was more storage. Because we were using dc2.8xlarge clusters, this meant adding more nodes of dc2.8xlarge instance types to the Amazon Redshift clusters. However, due to a rapid increase in our user participation over the last few years, we observed that our data volumes increased multi-fold. Initially, we were analyzing upwards of terabytes of data per day with Amazon Redshift clusters that ran mainly on dc2.8xlarge nodes. In near-real time, we analyze various aspects that directly impact the end-user experience, which can have a profound business impact for Dream11. The Dream11 Data Engineering team runs the analytics pipelines (what we call our Data Highway) across Dream Sports. We also look at why we were excited about this migration, the challenges we faced during the migration and how we overcame them, as well as the benefits accrued from the migration. In this post, we look at how we supercharged our data highway, the backbone of our major analytics pipeline, by migrating our Amazon Redshift clusters to RA3 nodes. Dream11 is the flagship brand of Dream Sports, India’s leading Sports Technology company, and has partnerships with several national and international sports bodies and cricketers. This is a guest post by Dhanraj Gaikwad, Principal Engineer on Dream11 Data Engineering team.ĭream11 is the world’s largest fantasy sports platform, with over 120 million users playing fantasy cricket, football, kabaddi, basketball, hockey, volleyball, handball, rugby, futsal, American football, and baseball. See for more information.ĪWS_REGION or EC2_REGION can be typically be used to specify the AWS region, when required, but this can also be defined in the configuration files.Post Syndicated from Dhanraj Gaikwad original
Modules based on the original AWS SDK (boto) may read their default configuration from different files. When no credentials are explicitly provided the AWS SDK (boto3) that Ansible uses will fall back to its configuration files (typically ~/.aws/credentials). If parameters are not set within the module, the following environment variables can be used in decreasing order of precedence AWS_URL or EC2_URL, AWS_PROFILE or AWS_DEFAULT_PROFILE, AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY or EC2_ACCESS_KEY, AWS_SECRET_ACCESS_KEY or AWS_SECRET_KEY or EC2_SECRET_KEY, AWS_SECURITY_TOKEN or EC2_SECURITY_TOKEN, AWS_REGION or EC2_REGION, AWS_CA_BUNDLE Controlling how Ansible behaves: precedence rules.Collections in the Theforeman Namespace.Collections in the T_systems_mms Namespace.Collections in the Servicenow Namespace.Collections in the Purestorage Namespace.Collections in the Openvswitch Namespace.Collections in the Netapp_eseries Namespace.Collections in the Kubernetes Namespace.Collections in the Junipernetworks Namespace.Collections in the F5networks Namespace.Collections in the Containers Namespace.Collections in the Cloudscale_ch Namespace.Collections in the Chocolatey Namespace.Collections in the Check_point Namespace.Virtualization and Containerization Guides.
0 kommentar(er)
