AWSでGPUも安く大量に使い倒せ
•Descargar como PPTX, PDF•
21 recomendaciones•11,902 vistas
2016/2/8に開催されたOpenCloudHPC #1 でのLT資料です。HPCといいながらGPU成分が多めです。
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a AWSでGPUも安く大量に使い倒せ
Similar a AWSでGPUも安く大量に使い倒せ (20)
Más de Yasuhiro Matsuo
Más de Yasuhiro Matsuo (14)
AWSでGPUも安く大量に使い倒せ
- 1. 1 OpenCloudHPC #1 AWSでGPUも安く大量に使い倒せ 2016年2月8日 アマゾン ウェブ サービス ジャパン株式会社 松尾康博
- 2. 2 Who am I ? • 名前 – 松尾康博 • 所属 – アマゾンウェブサービスジャパン株式会社 – ソリューションアーキテクト – 製造業のHPC、CAE、ビッグデータ解析等を担当 • 経歴 – 九州大学でスパコンの効率化研究 – SIerで 分散キューの開発・導入、分散処理研究 – Web系スタートアップCTO – SIerで仮想化基盤の研究・導入・運用 – 現職
- 4. 4
- 6. 6
- 9. 9 AWSとHPC
- 10. 10 Tightly Coupled (MPI/HPC) Requires Infiniband or other RDMA solution for scaling Loosely Coupled (Grid/HTC) Scales well with 10g Ethernet Data-Intensive Requires high-IOPS storage, or has very large datasets Data-Light Less dependence on high-IOPS, with smaller datasets Financial simulations Molecular modeling Contextual search Alt-coin mining Animation rendering Semiconductor verification Image processing / GIS Genomics Seismic processing High energy physics Metagenomics Human Brain Project Fluid dynamics Weather forecasting Materials simulations Crash simulations Grid Computing (“Pleasingly parallel”) Grid with IO Cluster Computing Cluster with IO (Data-intensive HPC) HPCワークロードとクラウド 得意 苦手
- 11. 11 Histogram of job sizes Workload Modeling for Computer Systems Performance Evaluation 著者: Dror G. Feitelson JOBサイズ毎の ジョブ数分布 JOBサイズ毎の CPU時間分布 スーパーコンピュータは 大規模ジョブに集中 手間のかかる小規模JOBは AWSで実行
- 13. HPC on AWSの事例
- 15. 15 Cycle Computing + Novartisの事例
- 16. 16 Amazon EC2 + Cycle Computing + HGSTの事例
- 18. 18
- 20. 20 Amazon EC2 スポットインスタンス 余っているEC2インスタンスを低価格で有効活用していただく仕組み 最大90%以上の割引価格でEC2インスタンスを使用可能 スポット入札アドバイザー、Spot Fleet、Spot Blockなどの 強力な周辺ツール/機能 分散処理、Workerが典型的なユースケース …だったが、新しい動きも出てきている Amazon EMR、Auto Scalingとの併用が容易 $1 $
- 21. 21 GPU インスタンスももちろん利用可能 g2.2xlarge E5-2670 2.6GHz 4core 15GB RAM 1x NVIDIA GPU 1536 Cores 4GB Mem g2.8xlarge E5-2670 2.6GHz 16core 60GB RAM 4x NVIDIA GPU 1536 Cores x 4 4GB Mem x 4 cg1.4xlarge X5570 8core 22.5GB RAM M2050 Fermi 448Cores 3GB ECC Mem 2010.11 2013.11 2015.4
- 23. 23 ap-northeast-1a (Tokyo Region) m4.large m4.xlarge スポット関連概念の整理 ap-northeast-1c m4.large … m4.xlarge c4.large 使用中 使用中 使用中 使用中 使用中 c4.large 使用中
- 24. 24 ap-northeast-1a (Tokyo Region) m4.large … m4.xlarge スポット関連概念の整理 - スポットプール c4.large ap-northeast-1c m4.large … m4.xlarge c4.large 使用中 使用中 使用中 使用中 使用中 使用中 Availability Zone（以下AZ）、OS、 インスタンスタイプごとの使われていないEC2インスタンスたち
- 25. 25 ap-northeast-1a (Tokyo Region) m4.large … m4.xlarge スポット関連概念の整理 - スポット価格 c4.large ap-northeast-1c m4.large … m4.xlarge c4.large 使用中 使用中 使用中 使用中 使用中 使用中 $0.0384 $0.0346$0.0346 $0.0530 $0.0209 スポットプール毎に需要と共有のバランスで変動する、その時点でのスポット インスタンス課金額 同じインスタンスタイプでもAZで異なる価格 $3.66
- 26. 26 ap-northeast-1a (Tokyo Region) m4.large … m4.xlarge スポット関連概念の整理 - 入札価格 c4.large ap-northeast-1c m4.large … m4.xlarge c4.large 使用中 使用中 使用中 通常 使用中 通常 使用中 通常 使用中 $0.0384 $0.0346$0.0346 $0.0530 $0.0209 「最大でここまでなら支払ってもよい」という価格 実際に課金されるのはスポット価格 管理コンソールまたはRequestSpotInstances APIから リクエスト可能 $3.66 「東京リージョンの 1aにあるc4.largeを 最大$0.05で使いたい！」
- 27. 27 ap-northeast-1a (Tokyo Region) m4.large … m4.xlarge スポット関連概念の整理 - 落札 c4.large ap-northeast-1c m4.large … m4.xlarge c4.large 使用中 使用中 使用中 通常 使用中 通常 使用中 通常 使用中 $0.0384 $0.0346$0.0346 $0.0530 $0.0209 入札価格がスポット価格を上回り、スポットプールに空きがあった場合※、 希望したスポットインスタンスを使いはじめることができる ※詳しくは「スポットインスタンスのしくみ」を参照のこと http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/how-spot-instances-work.html $3.66「東京リージョンの 1aにあるc4.largeは 現在$0.0346なので、 $0.05入札で起動できた！」
- 34. 34 EC2 各購入オプション料金比較例 2015年12月16日現在/東京リージョン/Linuxインスタンス。()内はOn-Demandからの節約比率 On Demand Reserved Instances for 1 year Spot Instance s Spot Block All Upfront Partial Upfront No Upfront 1h 6h c4.large $0.14 $0.0935 (33%) $0.095 (32%) $0.106 (24%) $0.0265 (81%) $0.077 (45%) $0.098 (30%) m4.large $0.183 $0.0961 (47%) $0.098 (46%) $0.115 (37%) $0.0206 (88%) $0.101 (44%) $0.128 (30%) r3.large $0.21 $0.1339 (36%) $0.1367 (35%) $0.157 (25%) $0.0202 (90%) $0.116 (44%) $0.147 (30%) 34
- 35. 35 オフピーク時間の定義は2015年12月16日現在の情報 Spot Block 1h 6h c4.large $0.070 (50%) $0.091 (35%) m4.large $0.093 (49%) $0.118 (35%) r3.large $0.107 (49%) $0.136 (35%) 35 リージョン オフピーク時間 (UTC) 米国東部 (バージニア北部) 土曜日 0:00～月曜日 0:00 米国西部 (北カリフォルニア) 土曜日 12:00～月曜日 12:00 米国西部 (オレゴン) 土曜日 12:00～月曜日 12:00 欧州 (アイルランド) 土曜日 9:00～月曜日 9:00 欧州 (フランクフルト) 土曜日 10:00～月曜日 10:00 アジアパシフィック (シンガポール) 土曜日 8:00～月曜日 8:00 アジアパシフィック (東京) 土曜日 9:00～月曜日 9:00 アジアパシフィック (シドニー) 土曜日 10:00～月曜日 10:00 南米 (サンパウロ) 金曜日 19:00～日曜日 19:00 オフピーク時間はさらに5%引き（スポットブロックのみ） https://aws.amazon.com/jp/ec2/spot/pricing/
- 38. 38 つまりスポットブロックとは 1. 最初に落札さえできれば、ブロック価格が高騰 しても課金額維持＆指定時間内は終了されない 2. 1〜6時間の短期間を前払いなしで割安利用でき るリザーブドインスタンスに近いもの • 途中で終了すれば課金も停止(期間縛りなし) • 価格はオンデマンドより安くスポットより高い というスポットインスタンスの拡張機能 38
- 40. 40 まとめ • AWSで使いたい新GPUインスタンスの仕様があ れば、フィードバックをお願いします！ g2.2xlarge E5-2670 2.6GHz 4core 15GB RAM 1x NVIDIA GPU 1536 Cores 4GB Mem g2.8xlarge E5-2670 2.6GHz 16core 60GB RAM 4x NVIDIA GPU 1536 Cores x 4 4GB Mem x 4 cg1.4xlarge X5570 8core 22.5GB RAM M2050 Fermi 448Cores 3GB ECC Mem
- 41. 41 最後に告知
- 45. 45
Notas del editor
- 20151009_HPC_on_AWS.pptx
- アマゾンでも、宅配の自動化でのDrone、エージェント機能をもつecho、気軽に買い物ができるDashというもをだしており、IoTを使った様々なチャレンジを行っており、また、IoTプラットフォームを提供している2lemetryという会社を買収しております。
- AWS Case Study: NASA JPL and Amazon SWF Do “Burns Cliff”, “Columbia Hills”, ”Endeavour”, and “Bonneville Crater” sound out of this world? You are right! These are some of the geographical structures visited by NASA’s Mars Exploration Rovers (MER). Over the years, the rovers beam back troves of exciting data including high-resolution images about the Red Planet. Now, Amazon Simple Workflow Service (Amazon SWF) joins some of the key computing technologies behind these missions, in enabling NASA scientists to reliably drive mission critical operations and efficiently process the growing knowledge we gather about our universe. NASA’s Jet Propulsion Laboratory (JPL) uses Amazon SWF as an integral part of several missions including the Mars Exploration Rover (MER) and Carbon in the Arctic Reservoir Vulnerability Experiment (CARVE). These missions continuously generate large volumes of data that must be processed, analyzed, and stored efficiently and reliably. The data processing pipelines for tactical operations and scientific analysis involve large scale ordered execution of steps with ample opportunity for parallelization across multiple machines. Examples include generation of stereoscopic data from pairs of images, stitching of multi-gigapixel panoramas to immerse the scientist into the Martian terrain, and tiling of these gigapixel images so that the data can be loaded on demand. There is a global community of operators and scientists who rely on such data. This community is frequently on a tight tactical operations timeline, as short as only a few hours. To meet these needs, JPL engineers set a goal of processing and disseminating Mars images within minutes. The need for orchestration JPL has long been storing and processing data in AWS. Much of this work is done with the Polyphony framework, which is the reference implementation of JPL’s Cloud Oriented Architecture. It provides support for provisioning, storage, monitoring, and task orchestration of data processing jobs in the cloud. Polyphony’s existing toolset for data processing and analytics consisted of Amazon EC2 for computational capacity, Amazon S3 for storage and data distribution, as well as Amazon SQS and MapReduce implementations such as Hadoop for task distribution and execution. However, there was a key missing piece: an orchestration service to reliably manage tasks for large, complex workflows. While queues provide an effective approach to distribute massively parallel jobs, engineers quickly ran into shortcomings. The inability to express ordering and dependencies within queues made them unfit for complex workflows. JPL engineers also had to deal with duplication of messages when using queues. For example, when stitching images together, a duplicate task to stich an image would result in expensive redundant processing the result of which would drive further expensive computation as the pipeline proceeds to completion in futility. JPL also has numerous use cases that go beyond brute data processing and require mechanisms to drive control flow. While engineers were able to implement their data driven flows easily with MapReduce, they found it difficult to express every step in the pipeline within the semantics of the framework. Particularly, as the complexity of data processing increased, they faced difficulties in representing dependencies between processing steps and in handling distributed computation failures. JPL engineers identified the need for an orchestration service with the following characteristics: Highly Available: To support mission critical operations Scalable: To facilitate parallel execution simultaneously from hundreds of Amazon EC2 instances Consistent: A scheduled task must be executed once with very high probability Expressive: Simple expression of complex workflows to expedite development Flexible: Workflows execution must not be limited to Amazon EC2 and tasks must be routable Performant: Tasks should be scheduled with minimal latency JPL engineers used Amazon SWF and integrated the service with the Polyphony pipelines responsible for data processing of Mars images for tactical operations. They gained unprecedented control and visibility into the distributed execution of their pipelines. Most importantly, they were able to express complex workflows succinctly without being forced to express the problem in any specific paradigm. Amazon SWF for MER and MSL To support the Fast Motion Field Test for the Curiosity rover, also known as Mars Science Laboratory, JPL engineers had to process images, generate stereo imagery, and make panoramas. Stereo imagery requires a pair of images acquired at the same time, and it generates range data that tells a tactical operator the distance and direction from the rover to the pixels in the images. The left and right images can be processed in parallel; however, stereo processing cannot start until each image has been processed. This classic split-join workflow is difficult to express with a queue based system, while expressing it with SWF requires a few simple lines of Java code together with AWS Flow Framework annotations. Generation of panoramas is also implemented as a workflow. For tactical purposes, panoramas are generated at each location where the rover parks and takes pictures. Hence, anytime a new picture arrives from a particular location, the panorama is augmented with the newly available information. Due to the large scale of the panoramas and the requirement to generate them as quickly as possible, the problem has to be divided and orchestrated across numerous machines. The algorithm employed by the engineers divides the panorama into large multiple rows. The first task of the workflow is to generate each of the rows based on images available at the location. Once the rows have been generated, they are scaled down to multiple resolutions and tiled for consumption by remote clients. Using the rich feature-set provided by Amazon SWF, JPL engineers have expressed this application flow as an Amazon SWF workflow. An Opportunity Pancam mosaic of total size 11280×4280 pixels containing 77 color images. Tiles at six levels of detail are required to deliver this image to a viewer at any arbitrary size. The yellow grid lines indicate the tiles required for each image. The Panoramas for Mastcam instrument on Mars Science Laboratory consist of up to 1296 images and have a resolution of almost 2 Gigapixels! The corresponding panoramic image is shown below. Orchestration in the cloud By making orchestration available in the cloud, Amazon SWF gives JPL the ability to leverage resources inside and outside its environment and seamlessly distribute application execution into the public cloud, enabling their applications to dynamically scale and run in a truly distributed manner. Many JPL data processing pipelines are structured as automated workers to upload firewalled data, workers to process the data in parallel, and workers to download results. The upload and download workers run on local servers and the data processing workers can run both on local servers and on the Amazon EC2 nodes. By using the routing capabilities in Amazon SWF, JPL developers dynamically incorporated workers into the pipeline while taking advantage of worker characteristics such as data locality. This processing application is also highly available because even when local workers fail, cloud based workers continue to drive the processing forward. Since Amazon SWF does not constrain the location of the worker nodes, JPL runs jobs in multiple regions as well as in its local data centers to allow for the highest availability for mission critical systems. As Amazon SWF becomes available in multiple regions, JPL plans to integrate automatic failover to SWF across regions. Beyond data processing pipelines JPL’s use of Amazon SWF is not limited to data processing applications. Using the scheduling capabilities in Amazon SWF, JPL engineers built a distributed Cron job system that reliably performed timely mission critical operations. In addition to the reliability, JPL gained unprecedented, centralized visibility into these distributed jobs through the Amazon SWF’s visibility capabilities available in the AWS Management Console. JPL has even built an application to backup vital data from MER on Amazon S3. With distributed Cron jobs, JPL updates the backups as well as audits the integrity of the data as frequently as desired by the project. All the steps of this application including encryption, upload to S3, random selection of data to audit, and actual auditing through comparison of onsite data with S3 are reliably orchestrated through Amazon SWF. In addition, several JPL teams have quickly migrated their existing applications to use orchestration in the cloud by leveraging the programming support provided through the AWS Flow Framework. JPL continues to use Hadoop for simple data processing pipelines and Amazon SWF is now a natural choice for implementing applications with complex dependencies between the processing steps. Developers also frequently use the diagnostic and analytics capability available through the AWS Management Console to debug applications during development and in tracking distributed executions. Using AWS, mission critical applications that previously took months to develop, test, and deploy, now take days.
- Nvidia GPUs (cg1.4xlarge) Intel Nehalem (cc1.4xlarge) 2TB of SSD 120,000 IOPS (hi1.4xlarge) Intel Sandy Bridge E5-2670 (cc2.8xlarge) Sandy Bridge, NUMA, 240GB RAM (cr1.4xlarge) 48 TB of ephemeral storage (hs1.8xlarge) 2.6 GHz Sandy Bridge CPU w/ Turbo enabled 1 NVIDIA GK104 GPU (Kepler) 8 vCPUs, 15 GiB of RAM 60GB SSD storage Ideally suited for remote desktop and 3D Supports DirectX, OpenGL, CUDA, OpenCL Wide range of platform partners including Citrix, Otoy, NICE Software
- 厳密にはネットワークタイプも。 どうでもいいけど、スポットプール、ぐぐると夏休みの海水浴スポットとかそんなのばっか出てくる
- 同じm4.xlargeでもAZが違うと価格が違う、と
- 同じm4.xlargeでもAZが違うと価格が違う、と
- くどいようだが課金されるのはスポット価格
- 効率よく＝より低コストまたは低リスクで。以前はこれをごりごり書いてる人が居た。さきほどの入札戦略の話で、価格履歴順応型というやつですね。 使い方は簡単です