My perspective

Hadoop - How it manages security

2011-11-28T06:10:00.001-08:00

Security is an important aspect of any IT deployment and Hadoop is no different. While running a large Hadoop cluster, it becomes increasingly important to authenticate users/services, prevent unauthorized access to data/services and isolate user Jobs so that only authorized users can submit, view and mutate job states. This article explains in detail the security aspects supported in Hadoop version 0.20.205.0.

Security in an IT setup typically involves following aspects

Authentication – Authentication is the process of verifying the identity of a particular user or service and ensuring user/service is really the one they are claiming to be. In Hadoop context services are NameNode, DataNode, JobTracker, TaskTracker & user proxy services like Oozie. Hadoop has pretty strong Authentication support through Kerberos, HMAC-SHA1 tokens and pluggable HTTP authentication.

Authorization - Authorization refers to granting or denying access to specific resources based on the requesting user's identity. This step is performed after a user is identified through authentication. Authorization is usually performed through access control lists, which associate user identities with specific rights. Hadoop supports authorization by limiting access to service APIs, HDFS files permissions, Job execution & administrative permissions and making Job information accessible only to authorized users

Data Integrity & Privacy – Data Integrity & Privacy refers to the ability of ensuring that data is not tampered with and sniffed when it is transferred over unsecure network channel. Hadoop has limited support for Data Integrity & Privacy. RPC calls with various Hadoop services can optionally be protected against tamper and sniff, however actual HDFS data/blocks are transferred unencrypted.

Players in Hadoop security –

User (access HDFS and Map-Reduce services)
HDFS and Map-Reduce services (services user requests and coordinates among themselves to perform Hadoop cluster operations)
Proxy service like Oozie (accesses Hadoop services on behalf of user)

Let us look in detail how various interactions among these roles are secured -

1. User to Service Authentication, Integrity and Privacy – Hadoop has following user to service interaction

a. User code interacts with Name Node for creation/deletion of files/directories, retrieve location of data blocks for file read/write operations and other file system administrative operations

b. User code interacts with Data Node to recover from block replication failure during file append

c. User code interacts with Data Node to read/write data blocks

d. User code interacts with Job Tracker to schedule job execution

e. Job tracker, Task trackers and Tasks interact with Name Node on behalf of a User to store and access job related files.

f. Job tracker, Task trackers and Tasks interact with Data Node on behalf of a User to store and access job related data blocks.

g. Tasks interact with Task Tracker on behalf of user to coordinate a Job’s tasks execution.

h. Reduce Tasks interact with Task Tracker through HTTP to download Map tasks output.

These interactions are depicted in picture below -

2. User Authorization

Service Level authorization - First level of authorization is service level access. All Hadoop services let you define the users and groups who are authorized to make RPC call to that service. Only if the user making RPC call belongs to authorized service user/group, the RPC call will go through. Checkout http://hadoop.apache.org/common/docs/stable/service_level_auth.html for configuration details of service level authorization

Authorization in HDFS - All files & directory created in Hadoop carries separate read and write permissions for user, groups and others (very much Unix like file permissions). Before allowing access to a file/directory, name node ensures that user has got appropriate permission for user/group/other based on logged in user credentials. Checkout http://hadoop.apache.org/common/docs/current/hdfs_permissions_guide.html for details.

Authorization in Map-Reduce – Job Tracker supports authorization on job queue, i.e. only users with appropriate authorization are allowed to submit, view, modify and cancel jobs. Checkout http://hadoop.apache.org/mapreduce/docs/current/mapred_tutorial.html#Job+Authorization & http://hadoop.apache.org/common/docs/current/cluster_setup.html for details. During job execution any access to HDFS made with user credentials. Only user with appropriate authorization can access HDFS files

3. Service-to-Service Authentication – Services are authenticated using Kerberos before they start communicating with other services. This authorization protects Services RPC APIs meant for Hadoop cluster management/coordination that are not directly connected to any user request (e.g. Name node and data node interact with each other for block management operations, TaskTracker interact with JobTracker for a Job’s task scheduling and status tracking). Following service-to-service authentication scenario exists in Hadoop-

a. Data Node and Name Node authenticates each other using Kerberos credentials

b. Job Tracker and Name Node authenticates each other using Kerberos credentials

c. Task Tracker and Job Tracker authenticates each other using Kerberos credentials

These interactions are depicted in picture below -

4. Service-to-Service Authorization – If Hadoop security is enabled, each service-to-service RPC interaction is already pre configured to allow access only to appropriate services. E.g. access to methods in ‘org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol’ (interface used by Data Nodes to interact with Name Node for data block management) is only allowed to services authenticated with data node principal.

5. Web UI Authentication, Integrity and Privacy – All Hadoop services (NameNode, DataNode, JobTracker, TaskTracker) exposes part of their functions and administrative interface through HTTP. Hadoop ships with inbuilt authentication to protect services HTTP interface through ‘SPNEGO Kerberos’ (users are authenticated using Kerberos credentials) or rudimentary authentication through user name defined in HTTP URL Parameters. Same user credentials setup for authentication with Hadoop service RPC interface works for HTTP Kerberos authentication. Additionally Kerberos HTTP authentication requires setting up Kerberos credentials for HTTP server running with in each service.

Hadoop HTTP Authentication can be extended to support other HTTP authentication mechanism as well.

Integrity and privacy can be enabled through SSL/TLS

6. Web UI Authorization – Works same as way as user authorization for HDFS and Map-Reduce (Read section 2b. and 2c. for details)

7. Proxy Services (e.g. Oozie) Authentication, Integrity and Privacy – Proxy services are configured as super-user with other Hadoop services. Proxy services authenticate themselves to core Hadoop services using Kerberos. Communication channel between proxy services and Hadoop service can optionally be secured for privacy and data integrity. Proxy service authentication requires setting up Kerberos credential for proxy service

Additionally proxy services can be configured to authenticate users before they use proxy service to access Hadoop infrastructure.

8. Proxy Service Authorization - Hadoop services allow super-users to perform operations on behalf of other non-super users and operations are authorized based the permission of non-super users. Since proxy services accesses the Hadoop cluster on behalf of other users, proxy services performs those operations as the user whom they proxy and hence authorized based on proxy user’s access rights mentioned in section 2b. and 2c.

In nutshell, Hadoop has strong support for authentication and authorization. On the other hand privacy and data integrity is optionally supported when Hadoop services are accessed through RPC and HTTP, while the actual HDFS blocks are transferred unencrypted. Hadoop assumes network involved in HDFS block transfer is secure and not publicly accessible for sniffing, which is not a bad assumption for private enterprise network.

Scaling Map-Reduce in new Hadoop release 0.23

2011-10-18T13:11:00.000-07:00

Scaling Map-Reduce in new Hadoop release 0.23 (Click to Download PDF)

With Read-Write hibernate 2nd level Cache strategy, why use transactional one at all

2009-07-28T05:50:00.000-07:00

Hibernate 2nd level cache is a great option for large performance improvements if configured correctly for a particular use case. While choosing a right 2nd level caching strategy applications typically battle between Read-Write and transactional strategy. Off late I have been trying to figure out what is in there Hibernate 2nd level transactional cache strategy implementation that is not offered by a read-write cache strategy implementation and what is the performance cost factor. And to my delight it turned out that a apart from offering a performance benefits most of the time, a properly implemented read-write cache (e.g. terracotta 2nd level hibernate cache) is functionally no less than a transactional cache except few caveats that either most of the application never run into or can be worked around easily if known what the caveats are. In this article I talk about the basic functioning of Hibernate read-write cache strategy (how does it guarantee transactionality), caveats and workarounds.

Read write caching strategy uses a combination of transaction time stamp, cache entry timestamp, cache entry version (if optimistic locking is used) and cache entry locking to guarantee repeatable read isolation level.

Some of the things hibernate does to ensure the liveliness of data read/updated inside a transaction is –

Every cache entry inserted/changed inside cache is assigned a timestamp
Every transaction gets a timestamp at start.
Whenever a cache entry is updated it is soft locked (in a distributed cache setup, the cache provider should have a cluster wide lock, otherwise cache correctness is compromised). ‘Soft lock’ is referred as ‘lock’ later and is a Hibernate specific state where cache entry is not available for read and other cache operations.
Once transaction completes, cache entry state is updated cache entry state along with the new time stamp information. The update entry is soft unlocked then. ‘Soft unlock’ is referred as ‘unlock’ later and is the normal cache entry state where it is available for read and other cache operations.

A transaction can only read those cache entries that

Have cache entry time stamp smaller than transaction timestamp (that i.e. cache entry reflects the database state prior to transaction start)
Not locked (i.e. it is not being updated by any other concurrently running transaction).

If any of the above 2 condition fails, then transaction is instructed to read from DB instead of cache and based on the configured DB database isolation level, transaction will read/block on appropriate DB entry. This is smart; in case of conflict you go to DB and let DB control the transaction conflict based on the DB semantics.

You might ask how Read-Write cache handles conflicting updates – i.e. two transactions updating same entry concurrently, how hibernate knows which update wins? Remember whenever an update is made to cache entry, it is locked. When a transaction tries to lock an already locked entry (i.e entry is already under update by another concurrently running transaction), hibernate keeps the entry locked even after completion of both transactions and any read for this particular cache entry is again deflected to DB. Later reads from the DB refreshes the cache entry state and unlocks the entry for other transactional reads/updates.

So far we talked about reading the correct state of entity. Now comes the big question – how is the cache transactionality/correctness maintained in Read-Write caching strategy during transaction commit, rollback & failures (the so called value proposition of transactional cache). Here is how -

When application commits transaction, cache entry is locked, there by deflecting all the reads for this entry to DB now.
Then changes are flushed to DB and transactional resources are committed.
Once transaction is committed (i.e. reflected inside DB), in the after completion phase cache entry is updated and unlocked. Any other transaction starting after the update time stamp can now read the cache entry contents.

This is what happens in different stages of transaction completion/failure -

So anytime lag between 2 & 3 (i.e. when DB and cache are out of sync), you are using DB to read the latest state.
If transaction rolled back, cache entry still remains locked and later reads from the DB refreshes the cache entry state
What if node making transactional change fails between step 2 & 3 (i.e. transaction is committed to DB but not to cache) and cache state is preserved (e.g. in clustered cache), is my cache left corrupted? not really. Since cache entry is locked, other transaction keep reading from the DB directly. Later hibernate times out the locked cache entry and its contents are refreshed with database state and cache entry is again back for read/write operations.

Do you still need a transactional cache that either integrates with hibernate native transactions or JTA? All a JTA transaction cache guarantees is cache state visibility across transactions and recoverability if any of the transaction phases fails. With read-write you are guaranteed to read the correct state all the time. If cache entry state gets inconsistent because of any failure in transaction commit phase, it is guaranteed to recover with correct state. This is all what a transactional cache guarantees, but at a higher cost (esp. when read outweigh writes).

Hibernate Read-Write cache strategy makes a smart decision of reading from the cache or database based on the cache entries. Anytime if cache can not guarantee the correct contents, application is deflected to DB.

What are the caveats?

Read-Write cache might compromise repeatable read isolation level if an entity is read from the cache and later its contents are flushed from 1st level (session) cache. If transaction reads the same entry again from DB later and in the meantime other transaction updated the entry state, current transaction will get different state than what it read earlier.

Note:

This should occur only if session cache contents are flushed otherwise once any entry is read from 2nd level cache/DB, every subsequent read in the same transaction will get the state from session cache and there by guaranteed the same state again and again. How many people really flush session cache?
If certain scenario demands a flush from session cache, it can be worked around by explicitly using hibernate LockMode.READ when loading the entity. 2nd level cache is by passed in this mode and data is read from DB.

Cache entries might expire in lock mode. In lock mode each entry is assigned a timeout value and if update doesn’t unlock the entry within specified timeout, the entry might be unlocked forcefully(this done to avoid any permanent fallout of cache entry from the cache, e.g. when node fails before unlocking the entry). A genuinely delayed transaction might create a very small window where cache contents are stale and other transactions are reading the old state. Cache entry timeout is a cache provider property and might to be able tune if provider supports it.

Note:

For this to occur, update has to be delayed, read has to occur after timeout and moreover the stale window is miniscule. So majority of the applications are safe anyways.

Entity type that are mostly updated and have got concurrent read and updates, read-write caching strategy may not be very useful as most of the read will be deflected to database.

While for majority of the application Read-Write cache is clear winner (repeatable read isolation level without overhead of cache provider maintaining explicit transactional boundaries), other cache modes (e.g transactional, non strict read write cache) might be used in some the apps depending upon the use.

To maintain transactional semantics in cluster, the cache provider has to support distributed locking along with other cache management features, which is one of the many Terracotta fortes. We at terracotta are releasing soon a highly optimized new version of hibernate 2nd level cache that supports all cache modes – Read-Write cache, transactional, not strict read write and read only. Check out www.terracotta.org for details.