Web Hosting Skills Rss Feed

Windows Azure Storage (WAS) - Architecture

Thu, 21 Mar 2013 09:14:01 PDT

Good afternoon, ladies and gentlemen!

WAS - cloud storage system that provides customers with the ability to store virtually unlimited amounts of data over any period of time. WAS was introduced in production-version in November 2008. Previously, it was used for internal purposes only Microsoft applications such as, for example, storing videos, music and games storage of medical records, etc. The article is based on work with the service repository and dedicated to the principles of operation of these services.

WAS customers have access to their data from anywhere at any time and pay only for what they use and store. WAS stored in data use both local and geographic replication to implement recovery after serious failures. At the moment, the repository WAS consists of three abstractions - blobs (files), Tables (structured storage) and queue (the message). These three data abstraction cover the need for different types of data storage for most applications. A common use case is to store data in blobs, queues using the same data is transmitted in these blobs, interim same data, status, and such temporary data stored in tables or blobs.

During the development of WAS were taken into account customer needs, and the most significant characteristics of architecture are:

Strong consistency - many customers want to have strict consistency, especially for corporate customers, which transfer to the cloud infrastructure. They also want to have the possibility to read, write, and delete under certain conditions for optimistic control strictly consistent data - for the Windows Azure Storage provides that the CAP-Theorem (Consistency, Availability, Partition-tolerance) is described as slozhnodostizhimoe at one point time: strong consistency, high availability, and Partition Tolerance.
Global and highly scalable namespace - for ease of use in the store WAS implemented global namespace, which enables you to store data and access them from anywhere in the world. Since one of the main goals of WAS is to allow storage of large volumes of data, a global name space must be able to address the exabytes of data.
Disaster recovery - WAS stores customer data in multiple data centers, which are located within a few hundred miles away from each other, and this redundancy provides effective protection against data loss due to a variety of situations, such as earthquakes, fires, tornadoes, and so on.
Multitenantnost and storage costs - reducing the cost of storage many customers are served from the one shared storage infrastructure, and WAS with this model, when the repository of many different clients with different storage capacity necessary for them grouped in one place, greatly reduces the total required amount of available storage than if WAS singled separate equipment for each customer.

Consider partitsiiruemoe more global namespace. A key goal of Windows Azure storage is to provide a single global name space, which would allow customers to place and resize any amount of data in the cloud. To provide a global namespace WAS uses DNS as part of the namespace, and the namespace is composed of three parts: the account storage partition name and object name.

Example:
http (s) :/ / AccountName..core.windows.net / PartitionName / ObjectName

AccountName - Store account name chosen by the client, is part of the DNS-name. This is used for the storage location of the main cluster, and, in fact, the data center, which stores the data and where to send all requests for data for this account. Client in one application can use multiple accounts names and store the data in a completely different field.

PartitionName - the name of the partition that defines the location of the data when queried cluster storage. PartitionName used to scale-up access to the data storage across multiple sites, depending on traffic.

ObjectName - if partition the set of objects to uniquely identify the object used ObjectName. The system supports atomic transactions for objects within a PartitionName. ObjectName value option for certain types of data PartitionName can uniquely identify the object within your account.

In WAS can be used to the full name of a blob blobs as PartitionName. For tables, be aware that each entity in the table has a primary key consisting of PartitionName and ObjectName, which allows you to group entities into one partition for an atomic transaction. Queues PartitionName value is the queue name, each message is placed in the queue has its own ObjectName, uniquely identifies the message within the queue.

Architecture WAS

Fabric-controller manages, monitoring, fault tolerance, and many other tasks in the data center. This mechanism, which is aware of everything that happens in the system, starting with a network connection and ending state of the operating systems in virtual machines. The controller maintains constant contact with their own agents, installed on the operating system and sending complete information on what is happening with this operating system, including OS version, service configuration, package configuration and so on. As for storage, the Fabric Controller allocates resources and manages the replication and distribution of data on disks, as well as load balancing and traffic. Windows Azure Storage architecture is shown in Figure 1.

Location Service (LS). This service manages all SS and namespace for accounts for all SS. LS distributes accounts for SS and implements load balancing and other management tasks. The very same service distributed over two geographically separated sites for own own safety.

Stream Layer (SL). This layer stores data on disk and is responsible for the distribution and replication of data across servers to store data within the SS. SL can be seen as a distributed file system layer within each SS, reads files (“streams”), how to store these files, replicate, and so on. The data is stored on the SL, but is available with Partition Layer. SL provides, in essence, an interface that is used only PL, and file system API, allows you to perform operations such as recording only Append-Only, which allows PL to open, close, delete, rename, read, add parts and combine large files ” streams “, ordered lists of large chunks of data, called” extents “(Fig. 2).

Stream can have multiple pointers to extents, and each extent contains a set of blocks. In this case, the extents can be “sealed» (sealed), that is, add to them new pieces of data impossible. If you are trying to read data from a Stream, the data will be recorded in succession from E1 to the extent extent at E4. Each stream is considered Partition Layer as one big file, and the content can be changed Stream or refer to the random-mode.

Block. The minimum unit of data available for reading and writing, which may be up to a certain N bytes. All recorded data is written to an extent in one or more of the combined blocks, the blocks do not have to be the same size.

Extent. Extents called units Replication Stream Layer, and the default Storage Stamp store three replicas for each extent, stored in the NTFS-file consists of blocks. The extent size used Partition Layer, 1 GB, while the smallest objects are complemented Partition Layer to one extent, and sometimes to a single unit. To store very large objects (such as blobs), the object is divided into several Partition Layer extents. Here, of course, Partition Layer ensures that, to what extent and what objects belong to the blocks.

Stream Manager (SM). Stream Manager monitors namespace stream-s, controls the status of all active stream and the extent and location between Extend Node, monitors the health of all Extend Node, creates and distributes extents (not blocks - they do Stream Manager does not know), and provides a lazy perereplikatsiyu extents replica was lost due to a hardware error or simply inaccessible and collects “garbage extents.” Stream Manager periodically polls and synchronizes the status of all Extend Node and extents that they hold. If SM reveals that the extent razreplitsirovan at less than the expected number of EN, SM produces perereplikatsiyu. The volume of the state if it can be called, can be small enough to fit in the memory of a Stream Manager. Only consumer and customer Stream Layer is Partition Layer, and they are so designed that they can not use more than 50 million and no more extents 100.000 Stream for one Storage Stamp (units are not taken into account, as they can be quite incalculable number) that quite fit in 32 GB memory Stream Manager.

Extent Nodes (EN). Each EN manages the repository for the replica set of extents assigned to them SM. EN N has attached disks, which are under the full control of it to maintain replicas of extents and blocks. In this case, EN does not know anything about the Stream (in contrast to the Stream Manager, who knows nothing about the blocks), and controls only extents and blocks that (extents) are, in fact, the files on the disk containing the data blocks and their checksums + card associations shifts in extents to the corresponding blocks and their physical location. Each EN has some idea of its extent and that, where the cues for specific extents. When specific extents no longer referenced by any one of the Stream-s, Stream Manager collects these “junk” and the extents of the need to notify the EN make room. The data in the Stream can only be added to the existing well data can not be modified. The add an atom - or the entire block of data is added or not added anything. At one point in time may be added to a few blocks in a single atomic operation, “adding a few blocks.” Minimum size, which is available to read from the Stream, is one unit. The operation is to add multiple blocks allows the customer to record large amounts of sequential data in a single operation.

Each extent, as has been said, has a definite ceiling on the size, and when it is full, the extent of the sealed (sealed) and further write operations operate on the new extent. In sealed extent can add data, and it is immutable (immutable).

There are some rules regarding the extents:

1. After adding the record and confirm the transaction to the client, all further reads that record from any replica must return the same data (immutable).
2. After sealing extent all read from any printed replica must return the same content extent.

For example, when the Stream is created, SM appoints first extent three replicas (one primary and two secondary) for the three Extent Nodes, which, in turn, selected SM for a random distribution between the different domains of updates and fixes and given the possibility of load balancing. Besides SM decides which replica will be Primary for all extents and write operations are performed first on the extent of the primary EN, and only then with the primary EN record takes two secondary EN. Primary EN and location of the three replicas does not change to an extent. When SM allocates an extent, the information is sent back to extent at the customer, who then knows what EN contain three replicas and which one is the primary. This information becomes part of the metadata Stream and cached on the client. When the last extent in the Stream is sealed, the process is repeated. SM allocates another extent, which now becomes the last extent in the Stream, and all new write operations are performed on a new last extent. To the extent each operation adding replicated three times on all replicas of the extent that the client sends all requests to write to the primary EN, but the read operation may be made to any replica, even unsealed extents. Append operation is sent to the primary EN, and primary EN is responsible for determining the shift in the extent and sequencing of all write operations in the event that occurs in parallel recording of one extent, sending an add with the necessary shift to two secondary EN and sending to the client for confirmation which is sent only when the operation was confirmed by adding the three replicas. If one of the replicas is not responding or is (or there was) any hardware error, an error is returned to the client record. In this case, the client communicates with the SM and the extent in which there is a write operation, sealed SM.

SM then allocates a new extent with the replicas on other available and EN notes this extent in the latter stream, and information about this back to the client, which continues to make an add in a new extent. It should be mentioned that the entire sequence for the sealing and placement of the new extent performed an average of only 20 milliseconds.
As for the process of sealing. In order to seal the extent, SM EN polls all three of their current length. In the process of sealing two scenarios - either all replicas of the same size, or some of the replicas is longer or shorter than the others. The second situation is only for error adding, when any of the EN (but not all) were not available. When sealing extent SM chooses the shortest length, based on the available EN. This allows you to seal the extents so that all changes are confirmed to the client, will be sealed. After sealing the extent confirmed by a length does not change and if the SM can not communicate with EN during sealing, but then becomes available to EN, SM EN this leads to the confirmation of the length of sync, leading to the same set of bits.

However, there may arise a different situation - SM can not communicate with EN, but Partition Server, which is a client, it can. Partition Layer, which a little later, has two modes of reading - reading records in certain positions and with the help of iterating through all the records in the stream. As to the first - Partition layer uses two types of Stream - a record and a blob. For these Stream reads always occur for certain positions (extent + shift length). Partition Layer performs the read operation for the two types, using the position information returned by a previous successful operation to add Stream Layer, which happens only when all three cues have reported that the operation succeeded added. In the second case, when all the entries in the Stream are moving sequentially, each partition has two separate Stream (metadata and log evidence) that Partition Layer will be read sequentially from beginning to end.

In the Windows Azure Storage introduced a mechanism to save on disk space and traffic without compromising the availability of data, and it is called erasure codes. The essence of this mechanism is that the extent is divided into N roughly equal-sized pieces (in practice, this is again a file), then the algorithm of Reed-Solomon codes added M fragments, correcting an error. What does this mean? Any X of N fragments equal in size to the original file to restore the original file is enough to collect all the fragments X and decode, and the remaining NX fragments can be removed, broken and so on. As long as the system is stored in more than fragments of M codes, error correction, the system can completely restore the original extent.

This optimization sealed extents very important when huge volumes of data stored in the cloud storage because it allows to reduce the cost of storage with three complete replicas of the original data source data to 1.3-1.5 depending on the number of fragments, and also extends the “sustainability” of the data compared with three replicas in storage Storage Stamp.

When performing write operations to the extent that has three replicas, all operations are put on performance with a certain time and, if the operation is not completed within this time, the operation should not be performed. If EN specifies that the read operation can not be fully implemented for some time, he immediately informs the customer. This mechanism allows the client to address with the read another EN.
Similarly, the data for which is used erasure coding - when a read operation does not have time to execute for the time period because of the workload, the operation can not be used to read the full piece of data, but it can take the opportunity to reconstruct the data and in this case refers to the read all the fragments in the extent to erasure code, and the first N responses will be used to reconstruct the desired fragment.
Drawing attention to the fact that the system can handle WAS very large Streams, may encounter the following situation: some physical drives serve and closed on serving large read or write operations from crop capacity for other operations. To prevent this, WAS does not assign the new disk I / O-operation when he was already scheduled for surgery that can be performed more than 100 milliseconds, or if already appointed operations were assigned, but not executed for 200 milliseconds.

When the data is defined as recorded Stream Layer, is an additional drive or a SSD as a storage for log of all entries in the EN. Disk logging completely allotted for one magazine, which consistently write operations are logged. When each EN performs the add operation, it writes the data to disk journaling and starts writing data to disk. If the disk journaling returned a successful operation before the data will be buffered in memory for as long as all the data is written to disk, all reads are served from memory. Disk usage logger provides important advantages, such as an add does not have to “compete” with the disk read data to confirm the transaction for the client. Journal allows operations to add Partition Layer to be more consistent and have less delay.

Partition Layer (PL). This layer contains special Partition Servers (daemon process) and is designed to manage your own abstractions storage (blobs, tables, and queues), namespace, order transaction, strict consistency of the objects, store data on SL and caching data reducing the number of I / O operations to disk. PL also involved partitioning of data objects within SS according PartitionName and further load balancing between servers partitions. Partition Layer provides an internal data structure called the Object Table (OT), which is a large table that can grow up to several petabytes. OT, depending on the load dynamically partitioned into RangePartitions and distributed to all in Partition Server Storage Stamp. RangePartition is a range of records in OT, starting with the smallest key is provided to the highest key.

There are several different types of OT:
• Account Table stores the metadata and configuration of storage for each account associated with the Storage Stamp.
• Blob Table stores all objects blobs for all accounts associated with Storage Stamp.
• Entity Table keeps all records of the entity account for all the storage associated with the Storage Stamp and is used to store table service Windows Azure.
• Message Table stores all messages for all queues for the storage of all accounts associated with Storage Stamp.
• Schema Table tracking scheme for all OT.
• Partition Map Table monitors all current RangePartitions for all Object Table and what Partition Server service which RangePartition. This table is used FE-servers to route requests to appropriate Partition Server.

Tables of all types have a fixed pattern that is stored in the Schema Table.
For all the schemes OT has a standard set of property types - bool, binary, string, DateTime, double, GUID, int32 and int64, in addition, the system supports two special properties and DictionaryType BlobType, the first of which allows for the addition of properties without specific schema as a record . These properties are stored in such a dictionary in the form of (Name, type, value). The second special feature is used to store large amounts of data and is currently used only for Blob Table, and the data blobs are not stored in the stream of records, and in a separate thread for the data blob is essentially the same stores only a reference to the data blob (list links “extent + shift length”). OT support standard operations - insert, update, delete, and read, as well as batch transactions for records with the same value PartitionName. Operations in a single package are confirmed as one transaction. OT also supports snapshot isolation to allow the read operation is carried out operations in parallel.

Partition Manager (PM) monitors and shares the big OT on N RangePartition within Storage Stamp and assigns certain RangePartition Partition Server. Information about what is stored where, stored in Partition Map Table. One RangePartition appointed one active Partition Server, which ensures that the two RangePartition not intersect.
Each Storage Stamp has multiple instances of PM and they “compete” for one Leader Lock, stored in Lock Service.

Partition Server (PS) services requests for RangePartitions, appointed by the server PM and stores all state partitions in Streams and manages the cache in memory. PS is capable of handling several RangePartition several OT, perhaps to an average of ten. PS serves the following components, keeping them in mind:
• Memory Table, version of the log evidence for RangePartition, containing all the recent changes that have not yet been confirmed by the reference point.
• Index Cache, cache, containing the reference point position data flow records.
• Row Data Cache, in-memory cache for data pages to record a checkpoint. This cache is only available for reading. When an access to the cache, checked Row Data Cache and Memory Table with preference to the latter.
• Bloom Filters - if the data is not found in the Row Data Cache and Memory Table, we examined the positions and checkpoints in the data stream, and their rough sorting is inefficient, and so for each control point using special bloom-filters that indicate whether it may be accessed to a record in the checkpoint.

Lock Service is used to select the service PM. Each PS also manages leasing with Lock Service to service partitions. On error, all PS N RangePartitions, serves this PS, forwarded available PS. PM picks N PS, based on their load, then PM appoints RangePartitions PS and updates Partition Map Table of relevant data that allows Front-End Layer find location RangePartitions, referring to Partition Map Table. RangePartition uses to store data Log-Structured Merge-Tree, each RangePartition has its own set of Streams in the Stream Layer and Stream relates entirely to certain RangePartition.

Each RangePartition can be one of the following Streams:
• Metadata Stream - Stream that is central to RangePartition. PM appoints PS partition, giving the name of the Metadata Stream PS.
• Commit Log Stream - Stream that is designed to store logs confirmed the insert, update, delete, applied to RangePartitions from the last point generated for RangePartition.
• Row Data Stream stores data records and position for RangePartitions
• Blob Data Stream used only for Blob Table to store the data blobs.
All of these are different Stream Stream to Stream Layer, controlled OT RangePartition. Each RangePartition in OT has only one data stream, eliminating the Blob Table - RangePartition Blob Table in a stream of data records to store the data for the last checkpoint record (position blob) and a separate data stream for a blob to store the data for a specific type BlobType.

To implement load balancing among Partition Servers and determine the total number of partitions in the Storage Stamp, PM holds three operations:
• Load Balancing. This operation is defined as a certain PS are overwhelmed, and then one or more RangePartitions reassigned to a less loaded PS.
• Split. This operation is defined when a certain RangePartition are overwhelmed, and this RangePartition divided into two or more smaller partitions, then these RangePartitions shared by two or more PS. PM sends a command to Split, but decides where the partition will be divided, PS, based on the AccountName and PartitionName. Thus, for example, to separate the two RangePartition B RangePartitions C and D, the following operations:
o PM sends a command to the PS division B to C and D.
o PS makes a reference point for B and no longer receive traffic.
o PS performs a special team MultiModify, collecting Streams with B (metadata, logs, and data verification) and creates a new set of Streams for C and D in the same order as in B (this happens quickly, as are, in fact, only pointers to the data). PS then adds new ranges Partition Key for C and D to the metadata.
o PS resumes service traffic for new partitions C and D.
o PS notifies the PM on the implementation of the division, updates Partition Map Table and metadata, then transfers the separated partitions for different PS.
• Merge. With this operation, the two “cold” or lightly loaded RangePartitions together so as to form a range of key issues in their OT. For this PM selects two RangePartitions with adjacent bands PartitionName, Low-load, and performs the following actions:
o PM carries C and D so that they are served by the PS, PS and tells the team to combine C and D to E.
o PS remains a reference point for C and D and briefly stops servicing traffic to C and D.
o PS MultiModify executes the command to create a new log file and confirm the data streams is each of these streams is the union of all the extents of the corresponding fluxes of C and D.
o PS creates a stream metadata E, containing the names of the log and confirm the data flow, the combined range of keys and indexes for E (extent + offset) for the log evidence (from C and D).
o begin service traffic RangePartition E.
o PM renews Partition Map Table and metadata.
For load balancing, track the following metrics:
• The number of transactions per second.
• Average number of pending transactions.
• Load CPU.
• The load on the network.
• Delay requests.
• Data Size RangePartition.
PM with controls heartbeat-om each of PS, and the information about it is passed back to the PM in response to heartbeat. If the PM sees RangePartition are overwhelmed (based metrics), he shared partition and sends a command to perform an operation PS Split. PS If he is experiencing a heavy load, but not RangePartition, then PM remaps available at this PS RangePartitions to less loaded PS. For load balancing RangePartition PM sends team PS, having RangePartition, to record the current checkpoint, after which the PS sends a confirmation PM and PM RangePartition overrides another and updates PS Partition Map Table.

The decision on the choice of partitioning mechanism based band (which employs RangePartition) instead of index-based hashes (when objects are assigned to the server via hash values of their keys) was justified by the fact that partitioning based on ranges helps easier to implement Performance Isolation, as the object of a specific account they are really close to the limits set RangePartitions, indexing based on a hash simplifies the task of the load on the server, but it denies the benefits of local objects to isolate and efficient transfer. Partitioning based on ranges can store objects of the same client together in one set of partitions, which also provides the ability to effectively limit or isolate potentially unsafe accounts. One of the drawbacks of such a campaign is to scale in the sequential access scenarios - for example, if the client writes all data to the end of the key-range table, then all write operations will be redirected to the latest RangePartition customer table. In this case, the advantage of partitioning and load balancing in the system is not used. If the client distributes write operations on a large number PartitionNames, the system quickly separates the table on the set RangePartitions and distributes them across multiple servers, which allows a linear increase efficiency.

Front-End (FE). Frontend layer consists of a set of stateless-server listens on. When prompted, FE reads AccountName, authenticate and authorize the request, and then converts it to the server partition to PL (Based on the PartitionName). Servers of the FE, cache, a card partitions (Partition Map), in which the system manages some PartitionName tracking ranges and which server partitions which PartitionNames serves.

Intra-Stamp Replication (stream layer). This mechanism controls the synchronous replication and data. He retains enough replicas on different nodes in different domains of errors to save the data in the event of an error, and it is performed entirely in SL. In the case of a write operation received from the client, it is confirmed only after a full successful replication.

Inter-Stamp Replication (partition layer). This replication is asynchronous replication between the SS, and he holds that replication in the background. Replication occurs at the object level, that is, either the whole object is replicated or replicates its changes (delta).

These mechanisms differ in that intra-stamp provides resistance against the “iron” errors that occur in large-scale systems, whereas the inter-stamp provides geographic redundancy against a variety of disasters that occur rarely. One of the main scenarios for this type of replication is geographic data replication between storage account data centers in order to recover from natural disasters.
All data in the services repository blobs and tables are replicated geographically (but turn - no). With geographically redundant storage platform saves again three replicas, but in two locations. When deploying a storage account LS Storage Stamp picks in each of the geographical locations and registers selected AccountName all Storage Stamp, with one of the locations will receive a “live” traffic, while the second, secondary, shall act as inter-stamp replication (essentially is the geographic replication). LS then updates the DNS for the new record AccountName.service.core.windows.net, leading to the VIP main location. Thus, if a datacenter something happens, data will be available from the second location. When the write operation goes to the main location for the storage account, the changes are replicated in full with intra-stamp replication in the Stream Layer, then the code of the successful completion of the operation is returned to the client. To confirm the operation is asynchronous replication occurs in a different geographic location and there is a transaction used on Partition Layer.

Regarding the geographical availability and how everything is restored in the event of major disruptions. If there was a serious failure in the main geographical location, it is natural that the corporation is trying to mitigate the effects of the maximum. However, if all the really bad and the data is lost, you may need to apply the rules of geographic fault tolerance - the customer is notified of the accident occurred in the main location, after which the DNS-record scrape the main location for the second (account.service.core.windows.net ). Of course, in the process of translation DNS-records hardly anything will work, but upon completion of existing blobs and tables are available on their URL. After completion of the transfer of the second geographic location rises in status to the host (as long as the failure does not happen once the data center into the ground). Also, immediately after the process of improving the status of the data center is initiated the process of creating a new second geographical location in the same region and further replication. The development team has been announced that users will be able to choose where it will be the second geographic location, in the event that one region is more than two data centers, but so far I have not noticed this (probably because I do not know of these regions).

Geographic replication process is much more interesting, if only because that our actions affect it more and more frequently than ephemeral dinosaur ate the data center, which led to a geo-failover.
So, for example, in our account, there are several stores blobs (example from Team blog), foo and bar. For the full name of the blob blob equal to PartitionKey. We perform two transactions A and B on the blob foo, then perform two transactions X and Y on the blob bar. The system ensures that a transaction will be geographically replicated before transaction B, and, accordingly, the transaction will be X geographically replicated before transaction Y. Otherwise, there are no guarantees - It is unknown how much time will be spent on geographical replication between transactions and transactions on foo to bar. Also, if at the time of replication of data center for some reason fails, it will be impossible to replicate the geographical recent transactions, it may happen that the replicated transaction A and X, whereas the B and Y transactions will be lost. Or replicated only A and B, X and Y will be gone. The same thing could happen with table service (given that the partition tables by a specified application PartitionKey entities, not the name of a blob).

Summary

Windows Azure storage services are a critical component of the platform, which provides services for the storage of data in the cloud and implement a combination of characteristics such as strong consistency global namespace, and high availability of data in multitenantnosti.

Main defender left the Adobe Flash in Apple

Thu, 21 Mar 2013 09:03:11 PDT

Adobe CTO Kevin Lynch went to work for Apple - a company that has blocked Flash on the iPhone, and probably accelerated the decline in the popularity of this technology, writes TechCrunch. Sam Lynch is known for having once wrote a blog Adobe emotional article in defense of Flash, although in recent years he was engaged in Adobe more promising projects.

Lynch has played an important role in promoting the cloud service Adobe Creative Cloud, as well as leading the development of Adobe Marketing Cloud. Both products have helped to move from Adobe packaged sales to cloud offerings with SaaS-approach to sales. In fact, Lynch has managed to turn a cloudy Adobe company faster than it could other technology companies.

Lynch also advocated the need for a multi-platform Adobe in the development and design of adaptive products. Thanks to him, Adobe has done a lot for the release of a full-featured products for mobile devices, including Adobe Photoshop Touch for iPad and Photoshop Touch for iPhone.

Apple made no secret of the fact that he considered the new center of the universe cloud user. iCloud, presented at the WWDC in 2011, was supposed to be just such a center for the ecosystem of Apple. The company has taken some steps for this, making it possible to synchronize your information and media on the iOS and OS X via iCloud. But at the same time, Apple is constantly faced with complaints about the performance of iCloud and iTunes Match.

Currently responsible for iCloud senior vice president of Apple Eddy Cue. Prior to that, he was engaged by the service predecessor MobileMe, whose reputation was even worse. It is likely that Q can benefit from the support of veterans of cloud services to address shortcomings iCloud.

Reportedly, Lynch as Vice President of Technology will report to Apple Senior Vice President of Technology Bob Mansfield. Mansfield deals with wireless and semiconductor technologies, but the name of his office gives him greater freedom of action. So if Apple is planning a big cloud breakthrough that Lynch could become a good candidate.

Virtual desktops on Linux

Thu, 21 Mar 2013 08:52:45 PDT

Hello!

Had this problem:
deployed on a server one powerful system with possible users of the system, like the desktop. Each user should be assigned a certain strictly defined configuration (for example, 1 core, 2GB operatives, etc.). Everything should be as transparent to the end user (connected, job, disconnected).

Number of users - not known (ie, at one point they may be one, or maybe 100). Should be something like a pool of dynamically extensible virtual machine to which users connect.
Wednesday all the same, users do not have to store the final VM.

Guglenie on this issue has shown that to solve this problem, Microsoft has Virtual Desktop Infrastructure (VDI) , and at VMware - View ( good description of what I want to get ).

Microsoft solution does not fit - the server VM and customers on Linux.
VMware solution conditionally suitable, but is unlikely to get it to use in our project because of a proprietary product.

Question: is there any software that can be as similar to deploy on Linux (RedHat)? KVM, Xen, VirtualBox, etc., interested in any ideas.

How to disable PCI-switches on the device on the fly

Thu, 21 Mar 2013 08:50:49 PDT

Hello

I am developing a manual test in which there is a need to simulate disabling SCSI fiber channel. I am developing a manual test in which there is a need to simulate disabling SCSI fiber channel.
Uses SCSI FC adapter QLogic ISP2532 coupled with OSes CentOS 6.2. There is some progress in research, at the moment I was able to cut off the card through the echo 0> / sys/bus/slot/2-3/power and bring it back through the echo> / sys / bus / pci / rescan. It seems to be all good, except for one. This works only once, that is, After connecting the device, disable it again does not work, and echo 0> / sys/bus/slot/2-3/power simply does not work. Retrip only after rebooting the entire operating system, which I absolutely do not fit. I guess there is a separate system module, which can be rebooted and it will give the desired yaffekt.

There are ideas, colleagues?
Advance inordinately.

Does ICANN technically possible to cancel delegation block ip addresses?

Thu, 14 Mar 2013 17:06:21 PDT

Question humanitarian.

Increasingly, in the articles on the state of information security, I find the claim of modern states based on ICANN. Since the implementation of many of the functions of the state (in management, defense, economy, etc.) are now going through the Internet, ICANN, «control the Internet,” theoretically could cripple the state fulfills a number of its functions.

It is clear that ICANN, manages the routing on the top level of the DNS, theoretically can not bypass a national domain. It is also clear that this problem is easily avoided by using a single gateway between devices inside and root server DNS. Well, citizens living abroad, and the government will not be able to receive services, but sovereignty over the flow of information within its territory the state does not lose.

However, ICANN’s still and allocates blocks ip addresses. In this connection, a question arises: whether it can select blocks ip addresses issued by a particular provider? Specifically, whether ICANN technically possible to select ip-address, which is currently running our portal of public services, and give it to someone else?

Do I understand correctly that even if ICANN would do it, as long as the server is on the network of public services, the new legal owner of ip address would be unable to join the network? And just because of some failure to shut down our server, again it this ip address to get to was not able to? Or is the system does not work?

How to install Python3.3 on Debian x64?

Thu, 14 Mar 2013 16:54:06 PDT

The situation: a server on a remote Debian 6 x64. There trying to make a Python script.
For the script needs Python 3.3 (this version, others do not roll) with the module sqlite3 (since the script writes to the database in this format)

On the server, the default installed python 2.7.

Managed to collect python3.3 from source.

But it is unclear how he collected sqlite3. Trying to install sqlite3:

 apt-get install sqlite3

leads to the fact that it is set for Python 2.7. That, in general, is to be expected.

Attempt to escape from a heap of different pythons on a single version of Python 3.3, without building from source failed. I read this article . Python found the correct version and under the right architecture

Prescribed repository python3.3 in / etc / apt / sources.list like this:

deb ftp.de.debian.org / debian Experimental main

Old updated on packages:

 apt-get update && apt-get upgrade

and when trying to install version 3.3:

 apt-get install python3.3

throws out a lot of missing dependencies in other packages:

How to solve the problem in the end?

OpenSUSE 12.3 released

Thu, 14 Mar 2013 16:52:26 PDT

Today, March 13, 2013, without delay was a new release of the popular distribution openSUSE 03/12 .
Of the innovations is worth noting:

Full transition to systemd, support journald.
UEFI support for x86_64, experimental support for Secure Boot.
Is under way to develop openSUSE 12.3 for ARMv7.
MariaDB instead of MySQL by default.

Just updated the following programs:

Linux 3.7.10.
KDE 4.10.
Gnome 3.6.3
Xorg 1.13
Mesa 9
PulseAudio 3

And, of course, not dull the new wallpaper . By the way, the dark color scheme is actually quite pleasant.
Also immediately with a clean install is possible to drive Memtest (available from GRUB2). In addition to the above, there are other innovations narrower character. For example, the first release, including a full range of OpenStack «Folsom» for cloud platforms. Also, now officially supported E17, Sawfish and Awesome. Also in the description pretty much told to increase the performance / completion of any part of the distribution (integration Bluetooth’a, YaST, PackageKit general rewrite, etc.). Despite the fact that the distribution switched to systemd, still available service management using files and / etc / init.d.

Traditionally, the installation package is available for i686 and x86_64 or Coordinated Universal 4,7 DVD image, or Live and KDE 4.10 or Gnome 3.6.3, or network installer. Live-images weigh slightly less than 1 GiB, which makes it impossible to write this image to a CD, but now on Live-image hold more programs (in particular, it has finally turned gparted).

Let us hope that the work done well, congratulations to all users of this great distribution with the new release.

Linux 3.8 Local Root Vulnerability

Thu, 14 Mar 2013 16:50:23 PDT

It seems that the call clone () with parameters CLONE_NEWUSER | CLONE_FS results in a uid 0, ie, provides a standard user root.
The exploit only works if the kernel includes support for namespaces, and the user has write access to the root file system (in a lot of root systems and home partition is on the same topic).

To launch the exploit in a 32-bit environment, change all occurrences of lib64 to lib, and ld-linux-x86-64.so.2 to ld-linux.so.2.

IP-address in the headers Google Apps

Thu, 14 Mar 2013 16:46:25 PDT

As everyone knows, when you send mail through GMail site in header indicates IP-address (usually a field called ” X-Originating-IP “or« Received »). Google explains this by the fact that the anti-spam filters enough internal GMail.

For Google Apps domains on this rule does not work, and in headings indicate X-Originating-IP, is not so in all domains - some address hiding. On Google and our support is not no clear explanation, in which cases the address indicated.

Possibly a IP-address to be useful, when Google Apps be able to use for free, but not so long ago, they closed a free registration, leaving only tariffs for business customers.

There are no representatives of Google? Explain, please, is it possible to disable the recording of this title, or to explain why and when it is recorded?

New distribution Backtrack: Meet Kali Linux

Thu, 14 Mar 2013 16:41:16 PDT

Known distribution for pentestinga BackTrack changes name to Kali Linux and moved to Ubuntu on Debian. Kali Linux is the best Linux distribution for penetration testing and security audit.

Kali is a complete reassembly BackTrack Linux, fully adhering to development standards Debian. The entire infrastructure has been revised, all instruments have been analyzed and packaged, also used Git.

More than 300 tools for penetration testing: After consideration of each instrument, which was included in BackTrack, was removed a large number of tools that either do not work or duplicate other instruments with similar functionality.

Kali Linux, like its predecessor, is a completely free and always will be. You never have to pay for Kali Linux.

Open source.

FHS compliant: Kali has been designed to adhere to Filesystem Hierarchy Standard, which allows all users to easily find the Linux binaries, support files, libraries, etc.

Extensive support for wireless devices.

Secure development environment: Kali Linux development team consists of a small group of trusted individuals who can write packages and interact with storage only when using multiple security protocols.

Multilingualism: Although tools for pentest, usually written in English, we have ensured that Kali is the real multi-language support, allowing most users to work in their native language and to find the tools you need for the job.

Fully customizable: We are fully aware that not everyone will agree with our design solutions, we have enabled our customers as easy as possible to configure Kali Linux of your choice, up to the core.

Support ARMEL and ARMHF: ARM-systems are becoming more and more common and affordable, resulting in a working installation for ARMEL and ARMHF systems. Kali Linux ARM repository is integrated with the main distribution as tools for ARM will be updated along with the other distributions. Cali is now available for the following ARM-devices: rk3306 mk/ss808, Raspberry Pi, ODROID U2/X2, MK802/MK802 II, Samsung Chromebook.

Simple Ways to Execute PHP in WordPress Posts and Pages

Thu, 01 Nov 2012 08:52:58 PDT

For those who need a custom solution

Since web design is comprised of several basic elements – HTML, CSS and PHP, which are 3 different languages, some people reach a different level of knowledge in them. On the other hand, systems like Wordpress are made to be used by people who don’t have knowledge for any of these languages. That’s the reason why directly inputting PHP code isn’t possible – because it wouldn’t be needed for the masses. Yet if you want to do it, there are ways, and that’s what we’re going to discuss that in this article.

Custom Template on Custom Page

First we’re going to discuss what Wordpress uses to display itself on the browser. Apparently, PHP does all the work in processing the content, and stores it so that it can be returned by certain functions in the theme template files. Once there, they’re displayed , wrapped around the HTML of the template. But what does that mean to us ?

One of the reasons why this is so important to us, is because we can use that structure to execute our own code . We can make a custom template to be used only on a certain page, where we need the PHP code executed.

Make a function in the current template

Another commonly used way to bring our own code into Wordpress is to build a function (or several functions) into our code. Using “if … else” conditionals, or “case” structures, we can execute the function with different parameters wherever we need it in the template. Wherever we need the code to run, we just execute the function like this :

<?php functionname($parametername) ?>

Both of the above mentioned ways have a problem, however. They cannot run in a specific post, because they depend on the template. We can either use them on the whole template, or on a single page that uses a different template, but not separately, for a post. So what do we do then ?

Use plugins that allow code execution

There are several plugins who do the same trick we need, but we’re going to focus on two particular plugins: Exec-PHP and Shortcode Exec-PHP. You’ve probably already figured out what the difference between them is.

Exec-PHP is a wordpress plugin which makes execution in posts as easy as 1-2-3. This can be handy for various reasons, especially if you have knowledge of how to use scripts or even write scripts professionally. You can use PHP libraries to generate graphics, calculate statistics and bring more unique content to your site.

Still, Exec-PHP has one crucial drawback – if you are running a blog with several authors and several people with access to posting ,security is rather compromised since anyone can write code in the posts.

Here’s where we introduce Shortcode Exec-PHP. This plugin has rights management, which allows us to define the level of rights necessary to write PHP code, which will later be included with a shortcode in the post. The advantage here is that only certain people can write the PHP scripts and snippets, but many can execute the shortcodes associated with them.

With Shortcode Exec-PHP you can have one administrator that writes the scripts, and several moderators who use them if they find necessary. That’s not only good for security, but also prevents unskilled people from executing code on Wordpress, and prevents potential data loss and crashes which may occur if an incompetent person was to attempt executing their code.

It doesn’t need to be hard to include your own functionality via PHP. As with many other things in Wordpress, it’s wonderful that someone has already thought about it, and has already made it easy via plugins.

OpenVZ: Mounting Host Devices/Partitions/Directories In A Container With Bind Mounts (Debian/Ubuntu)

Thu, 06 Sep 2012 15:47:56 PDT

Sometimes you are in a situation where you need to mount a hard drive, partiiton or directory from the OpenVZ host inside an OpenVZ container - for example, you add a fast SSD to the host and want to put your container’s MySQL databases on it to make MySQL faster. This tutorial explains how you can mount host devices/partitions/directories in an OpenVZ container with bind mounts.
1 Preliminary Note
In this tutorial I have an OpenVZ container with the container ID 101 which is running MySQL. I add a second hard drive to the host and want to put the container’s MySQL directory /var/lib/mysql on the second hard drive.
2 Preparing The Host
Host:

If you have added a new hard drive to the host, you should see it in the output of…

fdisk -l

... and you should see that it is unformatted (unless you have created partitions on it previously) - in this example the hard drive is /dev/sdb:

root@server1:~# fdisk -l

Disk /dev/sda: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00029d5c

Device Boot Start End Blocks Id System
/dev/sda1 * 1 3793 30461952 83 Linux
/dev/sda2 3793 3917 992257 5 Extended
/dev/sda5 3793 3917 992256 82 Linux swap / Solaris

Disk /dev/sdb: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdb doesn’t contain a valid partition table
root@server1:~#

Let’s create one single partition on it:

fdisk /dev/sdb

root@server1: ~# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel with disk identifier 0x31e0dc4b. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won’t be recoverable. Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to switch off the mode (command ‘c’) and change display units to sectors (command ‘u’). Command (m for help): <-- n Command action e extended p primary partition (1-4) <-- p Partition number (1-4): <-- 1 First cylinder (1-3916, default 1): <-- ENTER Using default value 1 Last cylinder, +cylinders or +size{K,M,G} (1-3916, default 3916): <-- ENTER Using default value 3916 Command (m for help): <-- t Selected partition 1 Hex code (type L to list codes): <-- L 0 Empty 24 NEC DOS 81 Minix / old Lin bf Solaris 1 FAT12 39 Plan 9 82 Linux swap / So c1 DRDOS/sec (FAT- 2 XENIX root 3c PartitionMagic 83 Linux c4 DRDOS/sec (FAT- 3 XENIX usr 40 Venix 80286 84 OS/2 hidden C: c6 DRDOS/sec (FAT- 4 FAT16 <32M 41 PPC PReP Boot 85 Linux extended c7 Syrinx 5 Extended 42 SFS 86 NTFS volume set da Non-FS data 6 FAT16 4d QNX4.x 87 NTFS volume set db CP/M / CTOS / . 7 HPFS/NTFS 4e QNX4.x 2nd part 88 Linux plaintext de Dell Utility 8 AIX 4f QNX4.x 3rd part 8e Linux LVM df BootIt 9 AIX bootable 50 OnTrack DM 93 Amoeba e1 DOS access a OS/2 Boot Manag 51 OnTrack DM6 Aux 94 Amoeba BBT e3 DOS R/O b W95 FAT32 52 CP/M 9f BSD/OS e4 SpeedStor c W95 FAT32 (LBA) 53 OnTrack DM6 Aux a0 IBM Thinkpad hi eb BeOS fs e W95 FAT16 (LBA) 54 OnTrackDM6 a5 FreeBSD ee GPT f W95 Ext'd (LBA) 55 EZ-Drive a6 OpenBSD ef EFI (FAT-12/16/ 10 OPUS 56 Golden Bow a7 NeXTSTEP f0 Linux/PA-RISC b 11 Hidden FAT12 5c Priam Edisk a8 Darwin UFS f1 SpeedStor 12 Compaq diagnost 61 SpeedStor a9 NetBSD f4 SpeedStor 14 Hidden FAT16 <3 63 GNU HURD or Sys ab Darwin boot f2 DOS secondary 16 Hidden FAT16 64 Novell Netware af HFS / HFS+ fb VMware VMFS 17 Hidden HPFS/NTF 65 Novell Netware b7 BSDI fs fc VMware VMKCORE 18 AST SmartSleep 70 DiskSecure Mult b8 BSDI swap fd Linux raid auto 1b Hidden W95 FAT3 75 PC/IX bb Boot Wizard hid fe LANstep 1c Hidden W95 FAT3 80 Old Minix be Solaris boot ff BBT 1e Hidden W95 FAT1 Hex code (type L to list codes): <-- 83 Command (m for help): <-- w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. root@server1:~#
Afterwards, there should be the partition /dev/sdb1

fdisk -l

root@server1:~# fdisk -l Disk /dev/sda: 32.2 GB, 32212254720 bytes 255 heads, 63 sectors/track, 3916 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00029d5c Device Boot Start End Blocks Id System /dev/sda1 * 1 3793 30461952 83 Linux /dev/sda2 3793 3917 992257 5 Extended /dev/sda5 3793 3917 992256 82 Linux swap / Solaris Disk /dev/sdb: 32.2 GB, 32212254720 bytes 255 heads, 63 sectors/track, 3916 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x31e0dc4b Device Boot Start End Blocks Id System /dev/sdb1 1 3916 31455238+ 83 Linux root@server1:~#
Let’s format it with ext4…

mkfs.ext4 /dev/sdb1

... and mount it to/mnt/sdb1

mkdir /mnt/sdb1
mount /dev/sdb1 /mnt/sdb1

You should see /dev/sdb1 in the output of…

mount

... now:
root@server1:~# mount /dev/sda1 on / type ext3 (rw,errors=remount-ro) tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) proc on /proc type proc (rw,noexec,nosuid,nodev) sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) udev on /dev type tmpfs (rw,mode=0755) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620) /dev/sdb1 on /mnt/sdb1 type ext4 (rw) root@server1:~#
To have /dev/sdb1 mounted automatically at boot time, add it to /etc/fstab:

vi /etc/fstab

[...]
/dev/sdb1 /mnt/sdb1               ext4    errors=remount-ro 0       1

3 Preparing The Container Container: As we want to move /var/lib/mysql to the new partition, we need to stop MySQL first, make sure it isn’t started automatically when the container is started, create a backup of /var/lib/mysql and a new, empty /var/lib/mysql directory:

/etc/init.d/mysql stop
update-rc.d -f mysql remove

mv /var/lib/mysql/ /var/lib/mysql_bak
mkdir /var/lib/mysql
chown mysql:mysql /var/lib/mysql
chmod 700 /var/lib/mysql

4 Creating A Bind Mount Script For The Container On The Host Host: On the host we now create the bind mount script /etc/vz/conf/101.mount (make sure you replace 101 with the correct container ID!) as follows:

vi /etc/vz/conf/101.mount

#!/bin/bash
. /etc/vz/vz.conf
. ${VE_CONFFILE}
SRC=/mnt/sdb1
DST=/var/lib/mysql
if [ ! -e ${VE_ROOT}${DST} ]; then mkdir -p ${VE_ROOT}${DST}; fi
mount -n -t simfs ${SRC} ${VE_ROOT}${DST} -o ${SRC}

You must adjust the SRC and DST variables to your needs. SRC is the directory which we want to mount in the container (/mnt/sdb1 in this example), and DST is the mount point in the container. If you want to add mount options like noatime, you can add the -o switch to the mount line, e.g. as follows:

#!/bin/bash
. /etc/vz/vz.conf
. ${VE_CONFFILE}
SRC=/mnt/sdb1
DST=/var/lib/mysql
if [ ! -e ${VE_ROOT}${DST} ]; then mkdir -p ${VE_ROOT}${DST}; fi
mount -o noatime -n -t simfs ${SRC} ${VE_ROOT}${DST} -o ${SRC}

Make the script executable…

chmod +x /etc/vz/conf/101.mount

... and restart the container (that’s why we disabled MySQL’s system startup links for the container in chapter three - /var/lib/mysql is empty right now which will result in MySQL errors):

vzctl restart 101

5 Using The Bind Mount In The Container Container: After the container restart is finished, you can log into the container and check if the new mount exists:

mount

root@test:~# mount /dev/simfs on / type simfs (rw,relatime) /dev/simfs on /var/lib/mysql type simfs (rw,relatime) proc on /proc type proc (rw,relatime) sysfs on /sys type sysfs (rw,relatime) tmpfs on /lib/init/rw type tmpfs (rw,nosuid,relatime,mode=755) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,relatime) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620) root@test:~#

df -h

root@test:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/simfs 1.0G 381M 644M 38% / /dev/simfs 30G 172M 28G 1% /var/lib/mysql tmpfs 128M 0 128M 0% /lib/init/rw tmpfs 128M 0 128M 0% /dev/shm root@test:~#
Go to the /var/lib directory:

cd /var/lib
ls -la

root@test:/var/lib# ls -la total 72 drwxr-xr-x 18 root root 4096 Jul 17 10:25 . drwxr-xr-x 13 root root 4096 Feb 13 2011 .. drwxr-xr-x 5 root root 4096 Jul 17 10:16 apt drwxr-xr-x 2 root root 4096 Oct 16 2010 aptitude drwxr-xr-x 2 root root 4096 Feb 13 2011 dhcp drwxr-xr-x 7 root root 4096 Jul 17 10:16 dpkg drwxr-xr-x 2 root root 4096 Jan 1 2011 initscripts drwxr-xr-x 2 root root 4096 Jan 1 2011 insserv drwxrwsr-x 2 libuuid libuuid 4096 Feb 13 2011 libuuid drwxr-xr-x 2 root root 4096 Apr 17 2010 logrotate drwxr-xr-x 2 root root 4096 Dec 14 2010 misc drwxr-xr-x 4 root root 4096 Jul 17 10:53 mysql drwx——— 3 mysql mysql 4096 Jul 17 10:24 mysql_bak drwxr-xr-x 2 root root 4096 Feb 13 2011 pam drwxr-xr-x 2 root root 4096 Jul 17 10:23 quota drwxr-xr-x 2 root root 4096 Jul 17 10:52 update-rc.d drwxr-xr-x 2 root root 4096 Jul 17 10:53 urandom drwxr-xr-x 3 root root 4096 Feb 13 2011 vim root@test:/var/lib#
As you see, the new /var/lib/mysql directory has wrong permissions/ownership - let’s correct that (the new permissions/ownership should be kept even after a restart of the container):

chown mysql:mysql /var/lib/mysql
chmod 700 /var/lib/mysql

Now let’s move the databases from our backup back to /var/lib/mysql:

cp -pfr /var/lib/mysql_bak/* /var/lib/mysql

Then start MySQL:

/etc/init.d/mysql start

That’s it! One last thing you should do is recreate MySQL’s system startup links so that it starts automatically when the container is started:

update-rc.d mysql defaults

6 Links OpenVZ:http://openvz.org/