On Why I Don't Like Auto-Scaling in the Cloud - O'Reilly Broadcast
http://broadcast.oreilly.com/2008/12/why-i-dont-like-cloud-auto-scaling.html

Dynamic scaling is better than auto scaling in the cloud - George Resse

Good rant about why auto-scaling is not dynamic scaling: two of the most widely cited advantages of the cloud. Not entirely convinced by the argument, though. I suspect governors can be used to manage scaling in a sensible way.

Sometimes traffic is truly unexpected. But not as often as you think. If you know you are getting coverage in some publication, marketing should have done an ROI projection on the campaign and be able to provide you with expected response rates. But you don't want it to auto-scale. Auto-scaling cannot differentiate between valid traffic and non-sense. You can. If your environment is experiencing a sudden, unexpected spike in activity, the appropriate approach is to have minimal auto-scaling with governors in place, receive a notification from your cloud infrastructure management tools, then determinate what the best way to respond is going forward. Here, the auto-scaling is simply a band-aid to enable a human to use dynamic scaling to define an appropriate, temporary capacity to support the unexpected change in demand. If you know you have a batch window from midnight to 3am, set your cloud infrastructure management tools to add capacity at 11:30 and throttle back at 3:30.

Makes good sense. I'd probably disagree on a few points if the cloud is owned/controlled by one's own organization.

The SHOP.COM Cache System is an object cache system that...

leonard lin's weblog. topical geek news.

list of backend and frontend requirements

Learn everything you need to know about Scaling your Rails app through 13 informative Screencasts produced by Gregg Pollack with the support of New Relic.

Nice Screencasts series of performance and scalability of Rails Apps by Gregg Pollack

Scaling Rails screencasts produced by Gregg Pollack and supported by New Relic

Joe Stump, Lead Architect at Digg, gave this presentation at the Web 2.0 Expo. I couldn't find the actual presentation, but fortunately Kris Jordan took some great notes. That's how key moments in history are accidentally captured forever. Joe was also kind enough to respond to my email questions with a phone call.

Scaling Strategies

Scribe is a server for aggregating log data streamed in real time from a large number of servers. It is designed to be scalable, extensible without client-side modification, and robust to failure of the network or any specific machine.

Facebookのログ収集ソフトウェア

primer on ruby

Interesting article about MySQL scalability problems.

on-building-a-stupidly-fast-graph-database

connected to and things that connect to them. These are symmetrical — so creating a link from item A to item B, creates a reference from item B to item A.

aid, here is what it takes to do 10.000 gets and sets:

昨日のの邦訳

"Database sharding is the process of splitting up a database across multiple machines to improve the scalability of an application. The justification for database sharding is that after a certain scale point it is cheaper and more feasible to scale a site horizontally by adding more machines than to grow it vertically by adding beefier servers."

SELECT Name, Address FROM Customers WHERE CustomerID= ?", conn);

InfoQ: Rockstar Memcaching http://www.infoq.com/presentations/lutke-rockstar-memcaching performance presentation

About the conference RubyFringe is an avant-garde conference for developers that are excited about emerging Ruby projects and technologies. They're mounting a unique and eccentric gathering of the people and projects that are driving things forward in our community.

Memcached tutorial video from the RubyFringe conference.

Video of memcached best-practices

RubyFringe presentation on Memcached

tobias lutke rubyfringe talk about memcached

CollegeHumor

CollegeHumor memcache use

Redis is a key-value database. It is similar to memcached but the dataset is not volatile, and values can be strings, exactly like in memcached, but also lists and sets with atomic operations to push/pop elements.

“Redis is a key-value database. It is similar to memcached but the dataset is not volatile, and values can be strings, exactly like in memcached, but also lists and sets with atomic operations to push/pop elements. “In order to be very fast but at the same time persistent the whole dataset is taken in memory and from time to time and/or when a number of changes to the dataset are performed it is written asynchronously on disk. You may lost the last few queries that is acceptable in many applications but it is as fast as an in memory DB (beta 6 of Redis includes initial support for master-slave replication in order to solve this problem by redundancy).”

A nice fast K/V data store, with some nice list/set features.

After several iterations, false starts, and great conversations with Michael Nielsen, a flash of the obvious came: HTTP + Javascript! What if you could contribute to a computational (Map-Reduce) job by simply pointing your browser to a URL? Surely your social network wouldn't mind opening a background tab to help you crunch a dataset or two!

Concurrence is a framework for creating massively concurrent network applications in Python.

Concurrence is a framework for creating massively concurrent network applications in Python. It takes a Lightweight-tasks-with-message-passing approach to concurrency. The goal of Concurrence is to provide an easier programming model for writing high performance network applications than existing solutions (Multi-threading, Twisted, asyncore etc).

Concurrence is a framework for creating massively concurrent network applications in Python. It takes a Lightweight-tasks-with-message-passing approach to concurrency.

Super duper thingy

twitter tool

I am worried about Twitter. I love it the way it is today. But it's about to change big time, and I wonder whether it can survive the transition. Twitter …

We are on the edge of two potent technological changes: Clouds and Memory Based Architectures. This evolution will rip open a chasm where new players can enter and prosper. Google is the master of disk. You can't beat them at a game they perfected. Disk based databases like SimpleDB and BigTable are complicated beasts, typical last gasp products of any aging technology before a change. The next era is the age of Memory and Cloud which will allow for new players to succeed. The tipping point is soon. Let's take a short trip down web architecture lane: # It's 1993: Yahoo runs on FreeBSD, Apache, Perl scripts and a SQL database # It's 1995: Scale-up the database. # It's 1998: LAMP # It's 1999: Stateless + Load Balanced + Database + SAN # It's 2001: In-memory data-grid. # It's 2003: Add a caching layer. # It's 2004: Add scale-out and partitioning. # It's 2005: Add asynchronous job scheduling and maybe a distributed file system. # It's 2007: Move it all into the cloud. # It's 2008: Cloud +

What makes Memory Based Architectures different from traditional architectures is that memory is the system of record. Also discussed Jim Starkey NimbusDB

Plus a nice comment from Zed.

more application frameworks need a page like this. how to make it go. - added by harper reed's google reader

Really good talk summarizing Django best practices.

perfomance and scalability testing of a website

stress test your web app

What is Tsung ? The purpose of Tsung is to simulate users in order to test the scalability and performance of IP based client/server applications. You can use it to do load and stress testing of your servers. (Definition coming from the Tsung website) In this post, I will introduce the use of Tsung in order for you to stress test your web applications. Why Tsung ? Because it’s an Open-Source project and, to tell the truth, mainly because this application has been coded in Erlang which gives Tsung a little advantage on the other tools: it has the potential to simulate A LOT of concurrent requests … without crashing. That’s what we expect from a stress testing app, isn’t it? Let’s start the installation We will need the Perl Templating-Toolkit and the Gnu plotting utility in order to create nice HTML and graphical reports with the result data set. So, back to your command prompt: ~$ sudo apt-get install gnuplot-nox libtemplate-perl libhtml-template-perl libhtml-template-expr-pe

ウェブアプリケーションのパフォーマンステスト用ツール Tsung の使い方

There's a growing trend to provide some pretty awesome IT services over the internet. Seems to me that's the way it will mostly be in a decade's time - or less.

Amazon Elastic MapReduce is a web service that enables businesses, researchers, data analysts, and developers to easily and cost-effectively process vast amounts of data. It utilizes a hosted Hadoop framework running on the web-scale infrastructure of Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3).

Who needs infrastructure? Keep your data somewhere else. Process your data somewhere else. You can now run your small data business out of your garage. Just photoshop a nice office for the investors.

g system, mobile platform, or web platform. Basically, if you want to share a short thought, one to many, Twitter is a transport-independent way to do that. In a broader technical sense, we see ourselves as a short messaging layer for the internet. We’ve been described as a “telegraph for web 2.0.” One of the things that’s core to our business is providing open APIs for everything you can do on the website. So all the functionality that’s available there for users is also available for developers to access programmatically. That’s Twitter in a nutshell. Twitter started as a hack project at a company called ODEO, which was focused on podcasting. As ODEO was having some troubles in its latter days as a company, they started experimenting, to keep engineers involved by letting them play around with ideas they had on the side. One of the engineers, Jack Dorsey, had been really interested in status. He was looking at his AIM buddy list, and seeing that all of these guys were saying, “I’m w

How twitter uses scala to gain performance and stability

tags: scala twitter ruby programming scalability interview rails

"Steve Jenson: For example, if you make a change to your social graph; i.e., you follow or unfollow someone on Twitter. All of that work and the associated cache invalidations are done asynchronously by a daemon."

Three Twitter developers, Steve Jenson, Alex Payne, and Robey Pointer, talk with Bill Venners about their use of Scala in production at Twitter.

Cachr

"Facebook will complete its roll-out of a new photo storage system designed to reduce the social network's reliance on expensive proprietary solutions from NetApp and Akamai."

Experiences deploying a large-scale infrastructure in Amazon EC2 At OpenX we recently completed a large-scale deployment of one of our server farms to Amazon EC2. Here are some lessons learned from that experience. Expect failures; what's more, embrace them Fully automate your infrastructure deployments Design your infrastructure so that it scales horizontally Establish clear measurable goals Be prepared to quickly identify and eliminate bottlenecks As you carefully watch various indicators of your systems' health, be prepared to.... Play wack-a-mole for a while, until things get stable

ngine for Java makes it

exciting news!

Keeping your database simple and fast is often difficult if you use higher level frameworks such as ActiveRecords in Ruby or Java object persistence technologies such as Hibernate. There is a lot of magic that is happening out of sight that you have no control over. If you then have to scale your application it is often the relational database that these technologies require that becomes the performance and scaling bottleneck. Often requiring complex custom implementations of partitioning and sharding to make it work. The AWS services Amazon S3 and Amazon SimpleDB were designed to handle the dominant storage usage patterns within Amazon and they greatly reduced our need to rely on relational storage for scaling our systems. But it is almost never the case that a single storage technique is used in applications and services that need to operate at enterprise scale. For example it is a common pattern that objects stored in S3 using a primary key, have a collection of secondary keys (e.g

allthingsdistributed.com allthingsdistributed.com Database DatabaseSimpleandFast

facebook structure

TestSwarm is still a work in progress but I hope to open up an alpha test by the end of this month. Its construction is very simple. It's a dumb JavaScript client that continually pings a central server looking for more tests to run. The server collects test suites and sends them out to the respective clients.

Distributed Key Stores

(Anti RDBMS) Key-value stores

Nginx + php.

According to Gartner, a typical IT department spends 80% of their budget keeping the lights on, and this hampers their ability to drive change and growth in their business.

Google's definition of the "Cloud"

article about data architecture for facebook's photo system. Seems interesting

The Photos application is one of Facebook’s most popular features. Up to date, users have uploaded over 15 billion photos which makes Facebook the biggest photo sharing website. For each uploaded photo, Facebook generates and stores four images of different sizes, which translates to a total of 60 billion images and 1.5PB of storage. The current growth rate is 220 million new photos per week, which translates to 25TB of additional storage consumed weekly. At the peak there are 550,000 images served per second. These numbers pose a significant challenge for the Facebook photo storage infrastructure. NFS photo infrastructure The old photo infrastructure consisted of several tiers: * Upload tier receives users’ photo uploads, scales the original images and saves them on the NFS storage tier. * Photo serving tier receives HTTP requests for photo images and serves them from the NFS storage tier. * NFS storage tier built on top of commercial storage appliances. Since each ima

Since each image is stored in its own file, there is an enormous amount of metadata generated on the storage tier due to the namespace directories and file inodes. The amount of metadata far exceeds the caching abilities of the NFS storage tier, resulting in multiple I/O operations per photo upload or read request. The whole photo serving infrastructure is bottlenecked on the high metadata overhead of the NFS storage tier, which is one of the reasons why Facebook relies heavily on CDNs to serve photos. Two additional optimizations were deployed in order to mitigate this problem to some degree:

Excellent description of BASE design patterns

If ACID provides the consistency choice for partitioned databases, then how do you achieve availability instead? One answer is BASE (basically available, soft state, eventually consistent).

I've been reading a bunch of papers about distributed systems recently, in order to help systematize for myself the thing that we built over the last year. Many of them were originally passed to me by Toby DiPasquale. Here is an annotated list so everyone can benefit. It helps if you have some algorithms literacy, or have built a system at scale, but don't let that stop you.

Including transparent sending of production traffic to staging.

Maybe for Omniture testing?

Lift the curtain behind any modern web application and you will find at least a few proxy servers orchestrating the show. Caching proxies such as Varnish and Squid help us take the load of our application servers; reverse proxies such as Haproxy and Nginx help us partition and distribute the workload to multiple workers, all without revealing the underlying architecture to the user. In the Ruby world, Rack middleware and Rails Metal are sister concepts: both allow the programmer to inject functionality in the pre or post-processing step of the HTTP request.

Three clusters Production (Huge!!) Staging (one) Benchmarking (same as staging)

nice summary of different data stores...

Needle in a haystack: efficient storage of billions of photos

man, I really really wish I understood this stuff.

shows

“Django may be built for the Web, but CouchDB is built of the Web. I’ve never seen software that so completely embraces the philosophies behind HTTP. CouchDB makes Django look old-school in the same way that Django makes ASP look outdated.”

ebook on why you would choose couchdb

Check out this SlideShare Presentation : CouchDB: Perform like a pr0n star http://tinyurl.com/cukfou [from http://twitter.com/josefrichter/statuses/1588959474]

The first one is to use an in-memory data store, and communicate using the memcached protocol. This is, of course, *exactly* comparable to Memcached — behaviorally indistinguishable! — and it does worse. The second option is to do that, except switch to an on-disk data store. It’s pretty ridiculous that that’s still the same speed — communication overhead is completely dominating the time. Fortunately, Tyrant comes with a binary protocol. Using that substantially improves performance past Memcached levels, though less than a direct in-process database. Yes, communication across processes incurs overhead. No news here, I guess.

"Tokyo Tyrant is a server implemented on top of Cabinet that implements a similar key/value API except over sockets. It’s incredibly flexible; it was very easy to run it in several different configurations. The first one is to use an in-memory data store, and communicate using the memcached protocol. This is, of course, *exactly* comparable to Memcached — behaviorally indistinguishable! — and it does worse. The second option is to do that, except switch to an on-disk data store. It’s pretty ridiculous that that’s still the same speed — communication overhead is completely dominating the time. Fortunately, Tyrant comes with a binary protocol. Using that substantially improves performance past Memcached levels, though less than a direct in-process database. Yes, communication across processes incurs overhead. No news here, I guess."

celery is a distributed task queue framework for Django. It is used for executing tasks asynchronously, routed to one or more worker servers, running concurrently using multiprocessing.

Not one of those "we're using hadoop, now we're cool" articles. Well written!

Hadoop

One MILLION connections !

And this is what we wanted a year ago. At least.

define

NOSQL debrief

braindump: NOSQL debrief

First ever meeting of the NoSQL community. Lists all the presentations that were given.

No to SQL? Anti-database movement gains steam

The meet-up in San Francisco last month had a whiff of revolution about it, like a latter-day techie version of the American Patriots planning the Boston Tea Party. The inaugural get-together of the burgeoning NoSQL community crammed 150 attendees into a meeting room at CBS Interactive. Like the Patriots, who rebelled against Britain's heavy taxes, NoSQLers came to share how they had overthrown the tyranny of slow, expensive relational databases in favor of more efficient and cheaper ways of managing data. "Relational databases give you too much. They force you to twist your object data to fit a RDBMS [relational database management system]," said Jon Travis, principal engineer at Java toolmaker SpringSource, one of the 10 presenters at the NoSQL confab (PDF). NoSQL-based alternatives "just give you what you need," Travis said. Open source rises up The movement's chief champions are Web and Java developers, many of whom learned to get by at their cash-strapped startups without Ora

The meet-up in San Francisco last month had a whiff of revolution about it, like a latter-day techie version of the American Patriots planning the Boston Tea Party.

piece on an alternative approach to data management

Cassandra is a hybrid non-relational database in the same class as Google's BigTable. It is more featureful than a key/value store like Dynomite, but supports fewer query types than a document store like MongoDB. Cassandra was started by Facebook and later transferred to the open-source community. It is an ideal runtime database for web-scale domains like social networks.

SQL Databases Are An Overapplied Solution (And What To Use Instead)

No, not SHARED lessons, I mean SHARD lessons. I have to admit that until about a year ago I didn't really know the term shards in relation to databases. Now don't confuse that with not understanding how databases can be horizontally scaled. I was introduced to that concept and helped to define the various ways it can be done but we just called it splits. Regardless of what you call it, there are some interesting challenges that are introduced. The well known challenges of consistency are discussed ad nauseam, even by me, so I'm not going there with this article. But besides that, there are some other lessons to learn when applying the pattern to your data.

Worth reading just for the section on intelligently designing shard counts. Great discussion on picking counts that smooth your cost step function

Stack Overflow Architecture | High Scalability

Stack Overflow is a much loved programmer question and answer site written by two guys nobody has ever heard of before. Well, not exactly. The site was created by top programmer and blog stars Jeff Atwood and Joel Spolsky. In that sense Stack Overflow is like a celebrity owned restaurant, only it should be around for a while. Joel estimates 1/3 of all the programmers in the world have used the site so they must be serving up something good.

Case study on Redis

Just wanted to add that even though there is a single point of failure the reliability due to software bugs has not been an issue and the dfs Namenode has been very stable. The Jobtracker crashes that we have seen are due to errant jobs - job isolation is not yet that great in hadoop and a bad query from a user can bring down the tracker (though the recovery time for the tracker is literally a few minutes). There is some good work happening in the community though to address those issues.

I few weeks ago, I posted about a conversation I had with Jeff Hammerbacher of Cloudera, in which he discussed a Hadoop-based effort at Facebook he previously directed. Subsequently, Ashish Thusoo and Joydeep Sarma of Facebook contacted me to expand upon and in a couple of instances correct what Jeff had said. They also filled me in on Hive, a data-manipulation add-on to Hadoop that they developed and subsequently open-sourced.

A talk given at OSCON 2009 on July 21st, 2009.

This tutorial examines how best to cope when the Real World intrudes on your carefully designed website.

There’s plenty of material (documentation, blogs, books) out there that’ll help you write a site using Django… but then what? You’ve still got to test, deploy, monitor, and tune the site; failure at deployment time means all your beautiful code is for naught. This tutorial examines how best to cope when the Real World intrudes on your carefully designed website.

An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads.

HadoopDB is: 1. A hybrid of DBMS and MapReduce technologies that targets analytical workloads 2. Designed to run on a shared-nothing cluster of commodity machines, or in the cloud 3. An attempt to fill the gap in the market for a free and open source parallel DBMS 4. Much more scalable than currently available parallel database systems and DBMS/MapReduce hybrid systems. 5. As scalable as Hadoop, while achieving superior performance on structured data analysis workloads

my students Azza Abouzeid and Kamil Bajda-Pawlikowski developed HadoopDB. It's an open source stack that includes PostgreSQL, Hadoop, and Hive, along with some glue between PostgreSQL and Hadoop, a catalog, a data loader, and an interface that accepts queries in MapReduce or SQL and generates query plans that are processed partly in Hadoop and partly in different PostgreSQL instances spread across many nodes in a shared-nothing cluster of machines. In essence it is a hybrid of MapReduce and parallel DBMS technologies. But unlike Aster Data, Greenplum, Pig, and Hive, it is not a hybrid simply at the language/interface level. It is a hybrid at a deeper, systems implementation level. Also unlike Aster Data and Greenplum, it is free and open source.

モバゲーのインフラの話

1日6億PVを6人のインフラ担当で支えられる理由

Intéressant, une étude des différentes db alternatives sous l'angle de la scalabilité

http://aws.amazon.com/s3/

data store scaling technologies

# Reverse Worker Code $worker= new GearmanWorker(); $worker->addServer(); $worker->addFunction("reverse", "my_reverse_function"); while ($worker->work()); function my_reverse_function($job) { return strrev($job->workload()); }

Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. It allows you to do work in parallel, to load balance processing, and to call functions between languages. It can be used in a variety of applications, from high-availability web sites to the transport of database replication events. In other words, it is the nervous system for how distributed processing communicates.

language independent worker framework

Riak combines a decentralized key-value store, a flexible map/reduce engine, and a friendly HTTP/JSON query interface to provide a database ideally suited for Web applications.

Google File System

ACM Queue, August 7, 2009

hadoop

Google and its MapReduce framework may rule the roost when it comes to massive-scale data processing, but there's still plenty of that goodness to go around. This article gets you started with Hadoop, the open source MapReduce implementation for processing large data sets. Authors Ravi Shankar and Govindu Narendra first demonstrate the powerful combination of map and reduce in a simple Java program, then walk you through a more complex data-processing application based on Hadoop. Finally, they show you how to install and deploy your application in both standalone mode and clustering mode.

Back in 2000, I went to France to build a genomics platform. A biotech hired me to combine their in-house genome data with that of public repositories like Genbank. The problem was the repositories, all with millions of records, each had their own format. It sounded like a massive, nightmarish data interoperability project. And an ideal fit for a hot new technology : XML

Three Rules for XML Rebels 1. Stop Inventing New Formats 2. Obey the Fifteen Minute Rule 3. Embrace Lazy Data Modeling

Un point de vue intéressant sur le xml, à rebours des conceptions en sciences de l'info (en tout cas les miennes)

Excellent thoughtful article on data bureaucracy and the limitations of XML.

ou combined memory

answer is 3TB database???

"The fundamental problem is endemic to the relational database mindset, which places the burden of computation on reads rather than writes."

Wow, cassandra uses a lot of disk space. Trade offs!

Casey: We’ve got 430,000 registered users, in a month we’ll see 200,000 of those, about 135,000 in a week and about 70,000 in a day. We peak at 3.6 million pageviews per day. That’s registered users only (doesn’t include the very few pages that are Google accessible) and does not include the usual API calls, RSS feeds, AJAX. Actual requests that hit Rails per day is 10 million. 900 new users sign up per day. The forums are very active with about 50,000 new posts being written each day. Some various numbers — 2.3 million knitting/crochet projects, 19 million forum posts, 13 million private messages, 8 million photos (the majority are hosted by Flickr).

Some various numbers — 2.3 million knitting/crochet projects, 19 million forum posts, 13 million private messages, 8 million photos (the majority are hosted by Flickr).

As a Web developer it's always a good idea to know what the current practices are in the industry even if they seem a bit too crazy to adopt…yet.

bit on why non-SQL dbs are used in social networking sites

Nice detailed examples on NoSQL data modeling in Cassandra.

am on the ga

Thursday morning's GDC Austin keynote was met with a large crowd as Blizzard Entertainment's J. Allen Brack and Frank Pearce took the stage to offer a detailed look into the inner workings of the genre-dominating World of Warcraft.

How Ravelry Scales to 10 Million Requests Using Rails

Interessantissimo articolo su un sito web fatto in rails (www.ravelry.com) che ha raggiunto volumi di traffico davvero ragguardevoli. L'articolo illustra come è nata l'idea per il sito (si tratta di un sito per gli appassionati di cucito e lavoro a maglia), come si è arrivati a quei numeri (10 million requests a day hit Rails, 3.6 million pageviews per day, 430,000 registered users. 70,000 active each day. 900 new sign ups per day), dell'architettura adottata, e delle lezioni imparate.

Tim Bray has a wonderful interview with Casey Forbes, creator of Ravelry, a Ruby on Rails site supporting a 400,000+ strong community of dedicated knitters and crocheters.

scalablity

A user named Apathy in this thread on how Reddit scales some of their features, shares some advice he learned while working at Google and other major companies. To be fair, I [Apathy] was working at Google at the time, and every job I held between 1995 and 2005 involved at least one of the largest websites on the planet. I didn't come up with any of these ideas, just watched other smart people I worked with who knew what they were doing and found (or wrote) tools that did the same things. But the theme is always the same: # Cache everything you can and store the rest in some sort of database (not necessarily relational and not necessarily centralized). How do you go about applying this strategy?

ing caches is a clasisc strategy for milking your servers as much as possilbe. First look for an exact match. If that's not foun

Cogent

Swarm is a framework allowing the creation of web applications which can scale transparently through a novel portable continuation-based approach. Swarm embodies the maxim "move the computation, not the data".

Facebook, Digg, Twitter が、なぜスケールするのに困難なのか？すべてはリアルタイムに更新されるデータだからという見解であっているかな？

とても勉強になる。

A software-based distributed caching system such as memcached is an important piece of today's largest Internet sites that support millions of concurrent users and deliver user-friendly response times. The distributed nature of memcached design transforms 1000s of servers into one large caching pool with gigabytes of memory per node. This blog entry explores single-instance memcached scalability for a few usage patterns.

"A software-based distributed caching system such as memcached is an important piece of today's largest Internet sites that support millions of concurrent users and deliver user-friendly response times. The distributed nature of memcached design transforms 1000s of servers into one large caching pool with gigabytes of memory per node. This blog entry explores single-instance memcached scalability for a few usage patterns."

Amazon Relational Database Service (Amazon RDS) is a web service that makes it easy to set up, operate, and scale a relational database in the cloud. It provides cost-efficient and resizable capacity while managing time-consuming database administration tasks, freeing you up to focus on your applications and business. Amazon RDS gives you access to the full capabilities of a familiar MySQL database. This means the code, applications, and tools you already use today with your existing MySQL databases work seamlessly with Amazon RDS. Amazon RDS automatically patches the database software and backs up your database, storing the backups for a user-defined retention period. You also benefit from the flexibility of being able to scale the compute resources or storage capacity associated with your relational database instance via a single API call. As with all Amazon Web Services, there are no up-front investments required, and you pay only for the resources you use.

Amazon Relational Database Service (Amazon RDS) is a web service that makes it easy to set up, operate, and scale a relational database in the cloud. It provides cost-efficient and resizable capacity while managing time-consuming database administration tasks, freeing you up to focus on your applications and business.MySQL

Amazon Relational Database Service (Amazon RDS) is a web service that makes it easy to set up, operate, and scale a relational database in the cloud. It provides cost-efficient and resizable capacity while managing time-consuming database administration tasks, freeing you up to focus on your applications and business. Amazon RDS gives you access to the full capabilities of a familiar MySQL database.

mysqlのレプリケーションを用いた構成について書いたblog記事。まだきちんと読んでいない。 フェールオーバー時のデータの完全性を保証しない構成。 想定する運用フローが書いてあるので、自分で考えるときの参考にする

Since 2005, VMware and Xen have gradually reduced the performance overheads of virtualization, aided by the Moore’s law doubling in transistor count, which inexorably shrinks overheads over time. AMD’s Rapid Virtualization Indexing (RVI – 2007) and Intel’s Extended Page Tables (EPT – 2009) substantially improved performance for a class of recalcitrant workloads by offloading the mapping of machine-level pages to Guest OS “physical” memory pages, from software to silicon. In the case of operations that stress the MMU—like an Apache compile with lots of short lived processes and intensive memory access—performance doubled with RVI/EPT. (Xen showed similar challenges prior to RVI/EPT on compilation benchmarks.)

Interesting set of links and posts describing the technologies Google builds its software on, and how they work together.

The Google Technology Stack … or as I would put it: An Introduction to MapReduce, Data Mining and PageRank

A great in-depth treament of the engine that powers Google

Part of what makes Google such an amazing engine of innovation is their internal technology stack: a set of powerful proprietary technologies that makes it easy for Google developers to generate and process enormous quantities of data. According to a senior Microsoft developer who moved to Google, Googlers work and think at a higher level of abstraction than do developers at many other companies, including Microsoft: “Google uses Bayesian filtering the way Microsoft uses the if statement” (Credit: Joel Spolsky). This series of posts describes some of the technologies that make this high level of abstraction possible.

Article summarizing presentation by Facebook on some of their scaling challenges and solutions.

This strategy is stated perfectly by Flickr's Myles Grant: The Flickr engineering team is obsessed with making pages load as quickly as possible. To that end, we’re refactoring large amounts of our code to do only the essential work up front, and rely on our queuing system to do the rest. Flickr uses a queuing system to process 11 million tasks a day. Leslie Michael Orchard also does a great job explaining the queuing meme in his excellent post Queue everything and delight everyone. Asynchronous work queues are how you scalably solve problems that are too big to handle in real-time.

Good introduction to the "NoSQL" space (initially not a fan of the term, but I guess it is going to stick...), highlighting the different designs used by the options in the space, and the benefits/drawbacks of those designs.

Unprecedented data volumes are driving businesses to look at alternatives to the traditional relational database technology that has served us well for over thirty years. Collectively, these alternatives have become known as “NoSQL databases.”

"Sharding kills most of the value of a relational database."

sql database db

Poor SQL; even with DSLs being the new hotness, people forget that SQL is one of the original domain-specific languages. It's a little verbose, and you might be bored with it, but it's much better than writing low-level mapreduce code.

Rails

"Twitter jilts Ruby for Scala" - Alex Payne http://tr.im/i7Vb [from http://twitter.com/kenmat/statuses/1436463196]

A nice overview of some of the more popular patterns in NoSQL architecture

Hazelcast is a clustering and highly scalable data distribution platform for Java. Features: Distributed implementations of java.util.{Queue, Set, List, Map} Distributed implementation of java.util.concurrency.locks.Lock Distributed implementation of java.util.concurrent.ExecutorService Distributed MultiMap for one-to-many relationships Distributed Topic for publish/subscribe messaging Transaction support and J2EE container integration via JCA Socket level encryption support for secure clusters Synchronous (write-through) and asynchronous (write-behind) persistence Second level cache provider for Hibernate Monitoring and management of the cluster via JMX Dynamic HTTP session clustering Support for cluster info and membership events Dynamic discovery Dynamic scaling Dynamic partitioning with backups Dynamic fail-over

Hazelcast is a clustering and highly scalable data distribution platform for Java.

Hazelcast is a clustering and highly scalable data distribution platform for Java. Features: * Distributed implementations of java.util.{Queue, Set, List, Map} * Distributed implementation of java.util.concurrency.locks.Lock * Distributed implementation of java.util.concurrent.ExecutorService * Distributed MultiMap for one-to-many relationships * Distributed Topic for publish/subscribe messaging * Transaction support and J2EE container integration via JCA * Socket level encryption support for secure clusters * Synchronous (write-through) and asynchronous (write-behind) persistence * Second level cache provider for Hibernate * Monitoring and management of the cluster via JMX * Dynamic HTTP session clustering * Support for cluster info and membership events * Dynamic discovery * Dynamic scaling * Dynamic partitioning with backups * Dynamic fail-over Hazelcast is for you if you want to * share data/state among many s

data distribution platform

Summary of the Facebook architecture and the bottlenecks they have had to work around

After considering a variety of data clustering algorithms, found that there was very little win for the additional complexity of clustering. So at Facebook, user data is randomly partitioned across indiviual databases and machines across the cluster. Hence, each user access requires retrieving data corresponding to user state spread across hundreds of machines. Intra-cluster network performance is hence critical to site performance. Facebook employs memcache to store the vast majority of user data in memory spread across thousands of machines in the cluster. In essence, nodes maintain a distributed hash table to determine the machine responsible for a particular users data. Hot data from MySQL is stored in the cache. The cache supports get/set/incr/decr and

Starting from Dynamo, ending with (roughly) follow @nosqlupdate on Twitter.

Materials that you need to read in order to get started with NoSQL

List of resources to read to get up-to-speed on the NoSQL movement.

If you're considering doing large scale analysis of structured data (access logs, for example), there are dozens of enterprise-level solutions ranging from specialized streaming databases, to the more mundane data warehousing solutions with star topologies and column store semantics. Google, facing the same problem, developed a system called Sawzall, which leverages their existing Map-Reduce clusters for large scale parallel data analysis by adding a DSL for easy manipulation of data.

Map/Reduce Toolkit by NY Times engineers is a great example of a Ruby DSL on top of the Hadoop Streaming interface. Specifically aimed at simplifying their internal log processing jobs, it exposes just the necessary bits for handling the access log inputs and provides a number of predefined reduce steps: unique, counter, etc. For example, to get a list of all unique visitor IP's, the entire program consists of:

Compare the recent sale of Friendster for a reported $26.4 million with Facebook’s projected 2010 revenues, of $1 billion, and we have a stark reminder of how the inability to scale can kill a startup. “All they had to do was keep the damned servers up and running,” Matt Cohler, a former Facebook executive and general partner at Benchmark Capital, says in Adam L. Peneberg’s book “Viral Loop,” but Friendster failed to scale and the cost was enormous.

Top 10 Internet Startup Scalability Killers – GigaOM

よくある構成っちゃ、よくある構成。MySQL ClusterとかMySQL Proxyとかの完成度があがっていくとこの部分の構成がもっとシンプルかつ強いシステムになっていくのかも

「dual master + ソフトウエアロードバランサ + Nagios」「masterサーバーに障害があった場合には、(Nagios のイベントハンドラで)それに直結する slave のヘルスチェック用ポートを閉じる」

事例

dual master, ソフトウェアロードバランサ, 仮想DNS, gethostbyname書き換え, masterが片方落ちたら直下のslaveもnagiosのイベントハンドラで落として一貫性保つ

How Google Works

'...while both [Google] search queries and processing power have gone up by a factor of 1000, latency has gone down from around 1000ms to 200ms. Crawler updates now take minutes compared to months in 1999.'

Configuring nginx as load balancer

Nginx

O mica documentatie cum sa faci un warm standby server PostgreSQL pentru HA ca sa replici o baza de date

restimate the impact that cloud computing is going to have on our industry. All of the sudden, the cost of hiring / fir

Typically excellent post from Ilya Grigorik.

An indepth look at disk i/o throughut, measurement and performance monitoring within Linux

あとで

I've been a big proponent of NoSQL for a while. I have played with just about all of the new generation of data stores. We almost got cassandra running in production once, and we've been running mongodb in production for about six months now. But, here's the thing: as awesome as these new dbs are, they're still young. Our app generates a ton of data and gets pretty serious traffic. So, we started hitting walls quickly. To make a long story short, we decided to fall back to MySQL. It's battle hardened. We know its production characteristics and limitations. Backups are a science. We know we can count on it. But, we have a lot of data, and adding fields and indexes was starting to get painful. Flexible schemas are one of the things that attracted me to NoSQL in the first place. Then, I remembered this article about How FriendFeed uses MySQL to store schema-less data. So, I decided to implement the system they describe in the article. Since we put Friendly in to production, we've seen

Friendly makes MySQL look like a document store. When you save an object, it seralizes all of its attributes to JSON and stores them in a single field. To query your data, Friendly creates and maintains indexes in separate tables. It even has write-through and read-through caching built right in.

Introducing Friendly: NoSQL With MySQL in Ruby Dec 16 2009 I've been a big proponent of NoSQL for a while. I have played with just about all of the new generation of data stores. We almost got cassandra running in production once, and we've been running mongodb in production for about six months now.

Note about Code-State-Cache-Data (CSCD) pattern in scalable web applications.

Short Article propounding the use of a "Code-State-Cache-Data-Architecture" (CSCD) instead of just CD or CCD applications. Basically saying that you should forget about stateful apps if you wan't maximum performance...

Application state - Data you can restore from the database or afford to lose if server is restarted (logged in users). He recommends storing this in-memory. "Application state goes into an in-memory key-value store like Tokyo Tyrant. Cache data goes into Memcached. Persistent data goes into a database"

"What he needs is the insight to identify state, cached data and persistent data in his application. Application state goes into an in-memory key-value store like Tokyo Tyrant. Cache data goes into Memcached. Persistent data goes into a database. Note that the seperation of code and application state may be beneficial later, because it allows you to scale easily by adding new memory servers. ... Let's call this the Code-State-Cache-Data (CSCD) pattern. What Damian originally had was a Code-Data (CD) pattern, and later he optimized to get a Code-Cache-Data (CCD) pattern"

Cloud computing means that the applications are running somewhere in the “cloud” (whether an internal network or the Internet). We, as users, don’t know and don’t care. Done right, cloud computing allows developers to develop, deploy and run applications that can easily grow in capacity (scalability), work rapidly (performance), and never – or at least rarely – fail (reliability), all without any concern as to the nature and location of the underlying infrastructure.

Cloud computing providersEveryone can build a website, but not not everyone can build a good and popular website. If you can pull it off, you can enjoy benefits like building something people are using, recognition and – sometimes – money. It can be a good business, even your full time job. But bad things can happen, and you should be prepared for it. One of those things (well, actually this is not a bad thing!) is to have spikes in traffic… maybe as a result of getting on Digg.com, Reddit.com, Google News, or another major media outlet frontpage… or getting reviewed by Techcrunch or another major technology blog.

I guess the conclusion is that there’s no use in preempting the technological progress of tomorrow. Machines will get faster and cheaper all the time, but you’ll still only have the same limited programming resources that you had yesterday. If you can spend them on adding stuff that users care about instead of prematurely optimizing for the future, you stand a better chance of being in business when that tomorrow finally rolls around.

From 37signals

VerkkoStadi Technologies is looking for a Hardcore PHP Developer. See more on the Job Board.

"The following information is meant to provide documentation on how others can recreate the benchmark trials used in our SIGMOD 2009 paper."

A Comparison of Approaches to Large-Scale Data Analysis: MapReduce vs. DBMS Benchmarks

import a lib in your python and run code automagically in parallel on a remote cluster. pricing is time used x processes plus data xfer

Mongo DB Production

Interesting blog post detailing production experiences with mongodb.

RT @kvz: Why Twitter is dropping MySQL in favor of Cassandra: http://bit.ly/dyeiXF

RT @DZone "Cassandra @ Twitter: An Interview with Ryan King « MyNoSQL" http://dzone.com/WbTY

MyNoSQL: Please include anything I’ve missed.

SQL is Scalable and NoSQL Isn’t For Everyone The point is one that I think all rational people already realize: The ACID RDBMS isn’t appropriate for every need, nor is the NoSQL solution.

"[Though as Michael Stonebraker points out, SQL the query language actually has remarkably little to actually to do with the debate. It would be more clearly called NoACID]"

5 Ways to Speed Up Your Rails App

trics on eBay’s main Teradata data warehouse include: * >2 petabytes of user data

Millions of queries per day

Statistieken over de databaseverwerking van ebay

une base données massivement parallèle et avec l'esprit "bigtable", provient de facebook

The Apache Cassandra Project develops a highly scalable second-generation distributed database, bringing together Dynamo's fully distributed design and Bigtable's ColumnFamily-based data model.

Java Platform (AKA, JVM) library/framework for distributed, fault-tolerant system through Actors. Scala and Java APIs. Found through Dean Wampler. Project tag line is "Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Actors"

Simpler Scalability, Fault-Tolerance, Concurrency & Remoting through Actors (Erlang con API Java e Scala)

# # Scaling practices turn a relational database into a non-relational database. To scale at Digg they followed a set of practices very similar to those used at eBay. No joins, no foreign key constraints (to scale writes), primary key look-ups only, limited range queries, and joins were done in memory. When implementing the comment feature a 4,000 percent increase in performance was created by sorting in PHP instead of MySQL. All this effort required to make a relational database scale basically meant you were using a non-relational database anyway. So why not just use a non-relational database from the start?

As Digg started out with a MySQL oriented architecture and has recently been moving full speed to Cassandra, his observations on some of their lessons learned and the motivation for the move are especially valuable. Here are some of the key takeaways you find useful:

RT @Sebdz: RT: @programmateur: Digg: 4000 % performance increase by sorting in PHP rather than MySQL (via @mrboo) - http://bit.ly/ckma10

♻ @n1k0: "Scaling practices turn a relational database into a non-relational database" http://n1k.li/4v (via @nsilberman)

Typically for relatively static data sets, relatively low query volumes, and relatively high latency requirements.

Today we’re announcing the availability of the Web-based AWS Management Console, which in this first release provides management of your Amazon EC2 environment via a point-and-click interface. A number of management tools already exist: for example a popular Firefox extension known as Elasticfox; however as you read more of this post I believe you’ll agree that the new console is compelling--especially when it’s time to log in as a new AWS developer.

These are all mysql oriented, but it sure seems like there are some fantastic principles to be pulling from here. Ex "covering indexes; orders-of-magnitude improvements." Or one on optimizing disk i/o.

Slides from a great variety of sessions @ Percona Performance Conference 2009 (April)

slides of the conference are available onliny

Rietveld

The second time through the Appistry team tossed the database, at least for its duties as a processing platform, instead keeping the transaction -- in fact ALL transactions -- in memory at the same time. This made the work flow into read-process-write (eventually). The database became more of an archive and suddenly a dozen commodity PCs could do the work of one Z-Series mainframe, saving a lot of power and money along the way.

twitter DB fan out　メール

TwitterのDB構成

Distributed systems are not typically a place where domain driven design is applied. Distributed processing projects often start with an overall architecture vision and an idea about a processing model which basically drives the whole thing, including object design if it exists at all. Elaborate object designs are thought of as something that just gets in the way of distribution and performance, so the idea of spending time to apply DDD principles gets rejected in favour of raw throughput and processing power. However, from my experience, some more advanced DDD concepts can significantly improve performance, scalability and throughput of distributed systems when applied correctly.

One of the most important building blocks of DDD that can help in distributed systems are aggregates. Unfortunately, at least judging by the discussions that I’ve had with client teams over the last few years, aggregates seem to be one of the most underrated and underused building blocks of DDD. I’m probably as guilty as anyone else of misusing aggregates and it took me quite a while to grasp the full potential of that concept. But from this perspective I think that getting aggregates just right is key to making a distributed system work and perform as expected.

Google Fellow Jeff Dean gave an excellent keynote talk at the recent WSDM 2009 conference that had tidbits on Google I had not heard before. Particularly impressive is Google's attention to detail on performance and their agility in deployment over the last decade.

HBase vs Cassandra: why we moved

Steve Huffman on Lessons Learned at Reddit By Keir Whitaker

7 Lessons Learned While Building Reddit to 270 Million Page Views a Month

Salesforce - incredible efficiency! and other metrics

all of Salesforce.com runs on only about 1,000 servers. And that is mirrored, so it is really only 500

a database implementing a dictionary, where every key is associated with a value. every single value has a type. The following types are supported: * Strings * Lists * Sets * Sorted Set (since version 1.1)

maybe the guy is not suitable to address such compare?

Persistent in-memory key value database compared to memcached

tructures and algorithms. Indeed both algorithms and data structures in Redis are properly choosed in order to obtain the best performance.

comet (and th

Java中性能赶超Erlang的NIO框架。

est was received it either got returned if it had data to return or stored with a timeout (the timeout was adde

A test with 200,000 sockets (note it's 100,000 pairs) showed a process size of 2MB, so far so good

bookmark してなかったのか… >そう考えると、私にはCreate/Update/Deleteのリクエストに対して、クライアントを待たせながら（つまり、HTTP Requestの処理に必要なスレッド・プロセスを保持したまま）データベースに変更をかけることが根本的に間違っているように思える。 これは同感なんだが、非同期にして comet 的に処理するとしても、他のリクエストとの整合性が必要なケースは存在するので、そこを確実にする配慮が必要になる筈。

問題の分割。実装詳細は詳しいのがほかにいくらでもあると思う

SCALABLE, OPEN-SOURCE SQL DBMS WITH ACID

A discussion of the different NoSQL-style datastores in use at Twitter, including Hadoop (with Pig for analysis), HBase,

Twitters NoSQL slides

A discussion of the different NoSQL-style datastores in use at Twitter, including Hadoop (with Pig for analysis), HBase, Cassandra, and FlockDB.

cassandra,thrift, hdfs, hbase, scribe,pig,lzo, flockdb

interesting presentation on #NoSQL at #twitter by @kevinweil http://bit.ly/99h8BK [from http://twitter.com/behi_at/statuses/13587582774]

536Share Exploring the software behind Facebook, the world’s largest site Posted in Main on June 18th, 2010 by Pingdom FacebookAt the scale that Facebook operates, a lot of traditional approaches to serving web content break down or simply aren’t practical. The challenge for Facebook’s engineers has been to keep the site up and running smoothly in spite of handling close to half a billion active users. This article takes a look at some of the software and techniques they use to accomplish that.

Software Behind Facebook

Interesting post about java and concurrency problems, with some nice comments too: http://bit.ly/dtzUL1 – Sergio Bossa (sbtourist) http://twitter.com/sbtourist/statuses/15701705927

270,000

Site speed is one of the most critical company goals for Facebook. In 2009, we successfully made Facebook site twice as fast, which was blogged in this post. Several key innovations from our engineering team made this possible. In this blog post, I will describe one of the secret weapons we used called BigPipe that underlies this great technology achievement.

Pretty cool way of building webpages from facebook to make it super fast.

BigPipe is a fundamental redesign of the dynamic web page serving system. The general idea is to decompose web pages into small chunks called pagelets, and pipeline them through several execution stages inside web servers and browsers. This is similar to the pipelining performed by most modern microprocessors: multiple instructions are pipelined through different execution units of the processor to achieve the best performance. Although BigPipe is a fundamental redesign of the existing web serving process, it does not require changing existing web browsers or servers; it is implemented entirely in PHP and JavaScript.

Site speed is one of the most critical company goals for Facebook. In 2009, we successfully made Facebook site twice as fast, which was blogged in this post. Several key innovations from our engineering team made this possible. In this blog post, I will describe one of the secret weapons we used called BigPipe that underlies this great technology achievement. BigPipe is a fundamental redesign of the dynamic web page serving system. The general idea is to decompose web pages into small chunks called pagelets, and pipeline them through several execution stages inside web servers and browsers. This is similar to the pipelining performed by most modern microprocessors: multiple instructions are pipelined through different execution units of the processor to achieve the best performance. Although BigPipe is a fundamental redesign of the existing web serving process, it does not require changing existing web browsers or servers; it is implemented entirely in PHP and JavaScript.

twice

SSL is not expensive

又拍网和大多数Web2.0站点一样，构建于大量开源软件之上，包括MySQL、PHP、nginx、Python、memcached、redis、Solr、Hadoop和RabbitMQ等等。又拍网的服务器端开发语言主要是PHP和Python，其中PHP用于编写Web逻辑（通过HTTP和用户直接打交道）， 而Python则主要用于开发内部服务和后台任务。在客户端则使用了大量的Javascript， 这里要感谢一下MooTools这个JS框架，它使得我们很享受前端开发过程。 另外，我们把图片处理过程从PHP进程里独立出来变成一个服务。这个服务基于nginx，但是是作为nginx的一个模块而开放REST API。

B上我们都建立了shard_001和shard_002两个逻辑数据库， Node-A上的shard_001和Node-B上的shard_001组成一个Shard，而同一时间只有一个逻辑数据库处于Active状态

又拍网和大多数Web2.0站点一样，构建于大量开源软件之上，包括MySQL、PHP、nginx、Python、memcached、redis、Solr、Hadoop和RabbitMQ等等。又拍网的服务器端开发语言主要是PHP和Python，其中PHP用于编写Web逻辑（通过HTTP和用户直接打交道）， 而Python则主要用于开发内部服务和后台任务。在客户端则使用了大量的Javascript， 这里要感谢一下MooTools这个JS框架，它使得我们很享受前端开发过程。 另外，我们把图片处理过程从PHP进程里独立出来变成一个服务。这个服务基于nginx，但是是作为nginx的一个模块而开放REST API。

B上我们都建立了shard_001和shard_002两个逻辑数据库， Node-A上的shard_001和Node-B上的shard_001组成一个Shard，而同一时间只有一个逻辑数据库处于Active状态

The goal of OpenStack is to allow any organization to create and offer cloud computing capabilities using open source software running on standard hardware. OpenStack Compute is software for automatically creating and managing large groups of virtual private servers. OpenStack Storage is software for creating redundant, scalable object storage using clusters of commodity servers to store terabytes or even petabytes of data.

via: http://news.ycombinator.com/item?id=1527572 What the software does: The goal of OpenStack is to allow any organization to create and offer cloud computing capabilities using open source software running on standard hardware. OpenStack Compute is software for automatically creating and managing large groups of virtual private servers. OpenStack Storage is software for creating redundant, scalable object storage using clusters of commodity servers to store terabytes or even petabytes of data.

The OpenStack Open Source Cloud Mission: to produce the ubiquitous Open Source Cloud Computing platform that will meet the needs of public and private cloud providers regardless of size, by being simple to implement and massively scalable.

The goal of OpenStack is to allow any organization to create and offer cloud computing capabilities using open source software running on standard hardware. OpenStack Compute is software for automatically creating and managing large groups of virtual private servers. OpenStack Storage is software for creating redundant, scalable object storage using clusters of commodity servers to store terabytes or even petabytes of data.

Open source hosting and storage software. database ?

How to process 135 million images in 9 days.

Yow. 135 million images resized.