odinserj

Hangfire Blog Moved

2015-04-14T00:00:00+00:00

Hi there! I’m writing just to inform you that Hangfire Blog moved to the official site (long time ago, sorry for the late notification):

https://www.hangfire.io/blog

I’m not’ ready to blog about anything else yet, and it would be very strange to write only about Hangfire in a personal blog. New product blog contains all the required links, has the same header as other parts of official site and thus is more useful.

Hangfire 1.3.0 Released

2014-12-10T00:00:00+00:00

Logging that just works

Logging was a very difficult thing to set-up in previous versions of Hangfire. Even if we omit tricky versioning issues, you had to install a separate logging adapter for your current logging library, with guessing the right one first (i.e. Common.Logging.NLog or Common.Logging.NLog20).

Now, the logging just works. If your project already uses Elmah, NLog, Log4Net, EntLib Logging, Serilog or Loupe you are not required to do anything. Thanks to awesome LibLog library by Damian Hickey (@randompunter on Twitter), installed logging library will be used automatically through the reflection.

If your application uses other logging library or a custom one, just implement two simple interfaces:

public interface ILog
{
    bool Log(LogLevel logLevel, Func<string> messageFunc, 
        Exception exception);
}

public interface ILogProvider
{
    ILog GetLogger(string name);
}

And yes, Hangfire is no more have Common.Logging or Common.Logging.Core dependencies.

Debugging with Symbolsource

SymbolSource is a service which exposes PDB symbols so you can step through Hangfire code while debugging your application. There are a few things to configure before we can begin, see symbolsource.org for more information.

Within Visual Studio:

Go to Tools → Options → Debugger → General
Uncheck “Enable Just My Code (Managed only)”
Uncheck “Enable .NET Framework source stepping”
Check “Enable source server support”
Uncheck “Require source files to exactly match the original version”
Go to Tools → Options → Debugger → Symbols
Select a folder for the local symbol/source cache
Add symbol servers under “Symbol file (.pdb) locations”: http://srv.symbolsource.org/pdb/Public

To speed up startup of the debug session, you can specify only the Hangfire dlls to be loaded. Go to Tools → Options → Debugger → Symbols, select “Only specified modules” and enter Hangfire.*.dll

Now you can step into Hangfire code while debugging.

Changes

Added – Generation of NuGet symbol packages with *.pdb and source files.
Added – Allow to customize serialization settings of Json.NET (by @fpellet).
Added – Ability to configure ServerTimeout option.
Breaking – Use LibLog package instead of Common.Logging for logging.
Changed – Hangfire.SqlServer is now merged with Dapper package.
Changed – Dapper package updated to 1.38.
Changed – Use ILMerge instead of ILRepack.
Changed – Update Microsoft.Owin.Host.SystemWeb to the latest version.
Fixed – Dashboard not crash if a scheduling task is cancelled (by @fpellet).
Other – Use psake instead of MSBuild for project build automation.

Introducing Hangfire Pro

2014-11-15T00:00:00+00:00

As an ASP.NET developer, I always wanted to have something simple to handle scenarios where you need to build a long-running async task and show a loader to a user:

Loading…
Nothing is being loaded here, don't let hypnotize yourself!

But every time I faced Windows Services, message queues and other difficult-to-understand problems and hard-to-maintain solutions, my head exploded into a billion pieces. At the same time I looked at Rails’ Sidekiq and wished to have a simple .NET alternative for background job processing. That is why I began to develop Hangfire (yep, just for the loader).

Slightly more than a year passed from the first commit and first version of Hangfire. This was my first open-source project that is being used more than in one organization, and it was an amazing experience for me to grow a project from scratch. I’m glad to see more and more people coming to the project and giving positive ratings and unvaluable feedback.

I want to keep Hangfire project as free as possible, but eliminate the most dangerous risk – the absence of time. There are many things to be done, including problems, new features, documentation, and I want to do them in a reasonable time.

Today, I’m introducing a new stage of Hangfire development – Hangfire Pro. It is a set of paid libraries that extend Hangfire functionality by providing features to improve performance and simplifying maintenance of background job processing in larger applications.

The Destiny of Hangfire.Redis

Bad News

First, I’ll start with bad news – Hangfire.Redis package is moved to Hangfire Pro. This means that it became available only for Pro users and will be removed from public NuGet Gallery soon.

I understand that this step breaks your expectations, and I’m sorry for this. The source code will be available at Hangfire.Redis repository, but there will be no updates and official support.

Other existing parts of Hangfire will not be moved to the Pro version ever. Moreover, I’ll try to do the vice versa, by open-sourcing paid extensions in the future, when the project will be more stable.

Good News

Most wanted features were implemented in the next version of Hangfire.Redis package. It now includes:

Compatibility with ServiceStack 4.0 libraries. It still uses ServiceStack v3 libraries, but ILMerge /internalize utility is being used to merge them into Hangfire.Redis assembly. So, you can use either Service v3 or v4 in your projects.
Configurable prefix for keys. You can use now the same database for different environments by setting corresponding prefix for keys in the RedisStorageOptions.Prefix property.

Hangfire Pro Today

Hangfire Pro already includes previously mentioned new version of Hangfire Redis package and a completely new Hangfire.PerformanceCounters package. The latter pushes internal metrics to Windows Performance Counters to proactively monitor issues with background job processing:

Hangfire Pro is available through the subscriptions you can buy at the official site. After purchasing a subscription, you will be able to do the following things:

Access to Hangfire Pro package binaries and sources on the private GitHub repository.
Access to priority and private support by dedicated e-mail (public forum discussions will be always free).
Use Hangfire under EULA for organizations that don’t want to use open-source licenses.

Hangfire Pro Roadmap

As I already said, Hangfire Pro will solve performance/usability/monitoring issues for larger web applications. Further work will be made in the following directions.

Batch Jobs

Batch jobs will help to build more complex workflows with Hangfire. They will give you the power of parallel processing and continuations, you can look at this code snippet:

BatchJob
    .Create(x =>
    {
        x.Enqueue(() => Console.Write("Messy"));
        x.Enqueue(() => Console.Write("Output"));
        x.Enqueue(() => Console.Write("With"));
    })
    .ContinueWith(() => Console.WriteLine("Predictable continuation"));

Three first tasks will be executed in parallel, end the continuation will be invoked only after successful run of the first tasks.

Async Methods Support

As with async controllers in ASP.NET MVC, this feature enables you to improve overall throughput keeping the lowest number of workers in the pool. Instead of waiting for an outstanding I/O operation, they will be able to process other background jobs as well.

public static async Task HighlightAsync(int snippetId)
{
    var snippet = await Context.Snippets.SingleOrDefaultAsync(snippetId);
    snippet.Code = await RemoteService.HighlightAsync(snippet.Code);

    await Context.SaveChangesAsync();
}

What are you waiting for?

Smart people say that content like this should contain a call to action. I have no enough experience to break the rules, so here it is:

Get a Subscription

Hangfire 1.0 Released

2014-06-30T00:00:00+00:00

Hangfire has finally reached the 1.0 milestone! This means that public API is frozen and considered to be stable. Starting from now, the SemVer 2.0 specification will be used for versioning every package (but there may be exceptions, follow the README for each package).

I want to thank everyone who helped me to reach this milestone, without your participation I would not be able to finish the work. Special thanks:

@devmondo – for endless optimism that helped me a lot in difficult times.
@dennyferra – for awesome Hangfire.RabbitMQ extension.

Release Notes

Added – OWIN implementation for Hangfire Dashboard.
Added – Custom authorization filters support for dashboard.
Added – Unified bootstrapper to start Hangfire in web applications.
Added – Descriptive names for background jobs in Dashboard.
Added – RabbitMQ support for SQL Server storage.
Changed – Hangfire now requires .NET Framework 4.5.
Changed – Namespaces, assemblies and packages now start with Hangfire (not HangFire).
Changed – Hangfire.Core now contains the dashboard. There is no more Hangfire.Web.
Changed – Common.Logging dependency package updated to 2.2.0.
Changed – Default dashboard url is now /hangfire (without .axd suffix).
Removed – Removed Enqueue method overloads with queue parameter.
Removed – Removed AspNetBackgroundJobServer class.
Fixed – Russian language in Dashboard replaced with English.
Fixed – Recurring jobs reported to be executed 44 years ago.

Upgrading from 0.x

If you have only Hangfire package installed, package update is very simple. Just type in your Package Manager Console window:

PM> Update-Package Hangfire

For custom installations, first update each package, and then remove the Hangfire.Web:

PM> Update-Package Hangfire.Core
PM> Uninstall-Package Hangfire.Web

Breaking Changes

Target your project to .NET Framework 4.5 or later.
Replace all occurrences of HangFire (case-sensitive) to Hangfire (lowercased f letter) in your projects.
Use the UseHangfire extension method for OWIN app builder instead of AspNetBackgroundJobServer.
Replace all links to Dashboard from /hangfire.axd to /hangfire.
Change your authorization rules as described here.

HangFire 0.9 Released

2014-06-07T00:00:00+00:00

Recurring jobs

This version brings incredibly easy method of scheduling and running recurring jobs inside ASP.NET applications. You need to call only a single line of code to perform this task:

RecurringJob.AddOrUpdate(() => Console.Write("Easy!"), Cron.Daily);

The Cron class provides different easy to use methods and overloads to set up recurring jobs on a minutely, hourly, weekly, monthly and yearly basis. It is also possible to use more complex CRON expressions, since the NCrontab library is being used to perform scheduling logic.

RecurringJob.AddOrUpdate(
    () => Console.Write("Powerful!"), 
    "0 12 * */2");

HangFire will check the schedule each minute and enqueue recurring jobs as regular background jobs, so you receive all power of HangFire, including the full transparency, free of charge.

HangFire Monitor was also updated and allows you to see and manage your recurring jobs:

So, HangFire now supports all kinds of background jobs: fire-and-forget, delayed and recurring, and let you process them with ease!

Changes

Added – Out-of-the-box support for running recurring jobs.
Added – SqlServerStorage now also accepts connection string names.
Changed – IBackgroundJobClient does not implement the IDisposable interface anymore.
Changed – IMonitoringApi does not implement the IDisposable interface anymore.
Changed – Improve DateTime resolution for job arguments by including milliseconds.
Changed – Removed deprecated RetryAttribute class. Please, use AutomaticRetryAttribute.
Fixed – KeyNotFoundException when accessing job cancellation token with Redis.
Fixed – Theoretical race condition that makes HangFire Server invisible from Monitor.

Upgrading

Version 0.9 brings some breaking changes. Please, do the following steps to perform the upgrade.

Replace the RetryAttribute with AutomaticRetryAttribute.
Remove the calls to Dispose method of IBackgroundJobClient interface (or BackgroundJobClient class).
Remove the invocations of IMonitoringApi.Dispose method.

HangFire 0.8.3 Released

2014-05-23T00:00:00+00:00

Cancellation tokens

This release comes with great feature for long-running jobs: cancellation tokens. They are like CancellationToken class, but for background jobs.

Previously, you was not able to stop a running job by using Requeue or Delete methods of the BackgroundJob class, or by clicking a button in HangFire Monitor. Its state was changed, but the job is still running. And running… And running… Nobody waits for it, why it is not stopped yet?

Job cancellation tokens provide ThrowIfCancellationRequested method that throws OperationCanceledException if a job was canceled due to:

Shutdown request – executed on BackgroundJobServer.Stop method invocation. Performed automatically when ASP.NET application is shutting down.
State transition – the state of the job has been changed, and it is not in the Processing state now. BackgroundJob.Delete, BackgroundJob.Requeue as well as all Monitor interface buttons lead to the job cancellation (in case it is running).

To use cancellation tokens, you need to add just one parameter of type IJobCancellationToken to a target method:

public static void Cancelable(
    int iterationCount, 
    IJobCancellationToken token)
{
    try
    {
        for (var i = 1; i <= iterationCount; i++)
        {
            // Loop breaker
            token.ThrowIfCancellationRequested();

            Thread.Sleep(1000);

            Console.WriteLine(
                "Performing step {0} of {1}...", 
                i, 
                iterationCount);            
        }
    }
    catch (OperationCanceledException)
    {
        Console.WriteLine("Cancellation requested, exiting...");
        throw;
    }
}

Then, create a background job:

BackgroundJob.Enqueue(
    () => Cancelable(1000, JobCancellationToken.Null)); 

To be able to test target methods, or to add the support of cancellation token to your old jobs, you can use the JobCancellationToken class:

public void Test()
{
    var token = new JobCancellationToken(true);
    Cancelable(10, token);
}

Hanging jobs

This release solved another problem. On some job queues (Redis, SQL Server, but not MSMQ) it is not possible to apply transaction just to fetch a job from a queue, hide it from another workers, and delete it on successful processing or place it back to a queue on failure or process termination.

To mimic this behavior, HangFire uses atomic get/set commands: update with output clause in SQL Server and BRPOPLPUSH in Redis. That is why other workers don’t see just fetched job. But in case of process termination, all jobs will remain in invisible state. To fight with this, there is a component on a server, who seeks invisible jobs and decides what to do with them.

Unfortunately, there is no way to determine whether a job was aborted by a process termination, or it is still working. To separate these cases, the component checks the fetching time of a job and compares it with current time. If the result is greater than InvisibilityTimeout, then we should trait the job as hanged, and return it to its queue. InvisibilityTimeout defaults to 30 minutes to be sure that the job processing was aborted.

When ASP.NET issues shutdown request and gives background jobs 30 seconds to die, some of them may be aborted by ThreadAbortException. In previous releases this lead to the fact that background job may stay in Invisible state on regular shutdown, and this sometimes increase the latency of job processing.

In this release HangFire make an attempt to place aborted job back to the start of its queue on this exception, and it will be started again immediately after application restart on success, and after InvisibilityTimeout on failure.

When the feature of re-queueing jobs on ThreadAbortException plays together with job cancellation tokens, it means that you greatly decrease the latency of background job processing, because the probability of using InvisibilityTimeout is greatly decreased.

TL;DR

Use job cancellation tokens where possible to ensure that your jobs are shutting down gracefully, and you greatly decrease the probability of high latencies in your background job processing.

Changes

Added – Cancellation token for job methods that throws on server shutdown and job aborts.
Added – Place interrupted job back to its queue if possible.
Fixed – Can not delete jobs when method or class was removed.
Fixed – NullReferenceException in Monitor.
Fixed – SqlException when changing state of a job with absent target method.

HangFire 0.8.2 Released

2014-05-21T00:00:00+00:00

Batch operations

Got tired to aim and click the Retry button on each failed job? It is much easier now, look at this:

Never knew that animated GIF optmimzation is so boring, especially for the first time.

You are also able to select all jobs in one click. Many thanks to GitHub Issues – they gave me an idea of how to make an ideal implementation of multiple items selection.

Additional metrics

When you write the code, it is important to have an instrument to measure the performance time. ASP.NET has different diagnostic tools for this task – Glimpse, MiniProfiler and other useful ones. But they are aimed to provide information about requests only, and almost useless for background jobs – they are being executed outside of a request.

I’ve implemented simple diagnostic feature (and not the replacement to full-stack performance profilers, such as dotTrace) for HangFire Monitor, and now you are able to see the following timings in HangFire Monitor:

Duration – job method + all filters performance time.
Latency – delay between the job creation and method invocation. This metric shows you background job invocation overhead.

As you can see, you can also watch the delays between state transitions. All timings have minimum resolution in 1 second. This resolution caused by using unix timestamps in HangFire. We need to change the things, but a bit later.

These metrics are added to the SucceededState, so they are also available to state filters.

DisableConcurrentExecution filter

This filter places a distributed lock in the beginning of a method performance and releases it after it was completed using the IServerFilter interface. The type and method name are being used as a locking resource. Each storage uses its own distributed lock implementation:

SQL Server uses sp_getapplock stored procedure;
Redis implementation uses the technique described in its documentation.

[DisableConcurrentExecution(timeoutInSeconds: 10 * 60)]
public void SomeMethod()
{
    // Operations performed inside a distributed lock
}

The filter is very useful in situations when you are accessing/changing shared resources that does not provide built-in locking routines. Otherwise you should make argumentative decision between using locking system provided by the resource itself, and a custom distributed locking like this filter.

For example, in SQL Server it is better to consider using different isolation levels and table hints first.

Changes

Added - Batch operations on jobs for HangFire Monitor.
Added - Retry and delete buttons for almost every page of HangFire Monitor.
Added - Duration and latency metrics for succeeded jobs.
Added - Display state transition latencies on job details page.
Added - DisableConcurrentExecution filter.
Misc - Tables in HangFire Monitor received some love.

HangFire 0.8.1 released

2014-05-17T00:00:00+00:00

Note. The following information may be outdated after 1.0 release. Please see the official documentation first.

Release notes

This release contains a bunch of new features, that complete the background job workflow (Deleted state), make it without additional latencies (MSMQ support) and inform you about failures in-time:

Added - MSMQ queues support for SQL Server job storage.
Added - “Deleted” state for jobs, when we don’t want to process them anymore.
Added - “Requeue” and “Delete” buttons on a job page in HF Monitor.
Added - Logging job failures: warning - there is a retry, error - no attempts left.
Added - BackgroundJob.Requeue and BackgroundJob.Delete methods.
Changed - Set InvisibleTimeout back from 5 to 30 minutes.
Changed - RetryAttribute is deprecated. Use AutomaticRetryAttribute instead.

As always, the new version of HangFire can be installed via NuGet Gallery. Here is the package list.

Deleting jobs

There is a new state – DeletedState, and some methods to perform the deletion – BackgroundJob.Delete and IBackgroundJobClient.Delete. When you are using these methods, a job is not being actually deleted, there is only state transition. Jobs in the deleted state expire after some delay (as succeeded jobs).

The operation does not provides guarantee that the job will not be performed. If you deleting a job that is performing right now, it will be performed anyway, despite of calls to delete methods.

Usage

If you want to delete job despite its current state, do the following:

var jobId = BackgroundJob.Enqueue(() => Console.WriteLine("Hello, world!"));
BackgroundJob.Delete(jobId);

// or
IBackgroundJobClient client = new BackgroundJobClient();
client.Delete(jobId);

If you want to be able to handle only deletion from exact state, for example, only from FailedState (because it may change after you click and before it will be deleted), you can specify it:

var jobId = BackgroundJob.Enqueue(() => Console.WriteLine("Hello, world!"));
BackgroundJob.Delete(jobId, FailedState.StateName);

// or
IBackgroundJobClient client = new BackgroundJobClient();
client.Delete(jobId, ScheduledState.StateName);

Manual deletion

Job details page now contains Requeue and Delete buttons (other pages also have these buttons). So, you can requeue and delete your jobs at any time:

MSMQ support for SQL Server storage

SQL Server job storage implementation does not require you to learn and install additional technologies, such as Redis, for projects to use HangFire. However, it uses polling to get new jobs that increases latency between the creation and invocation process (see also this feature request).

The MSMQ implementation, that was introduced in HangFire 0.8.1, replaces only the way HangFire enqueues and dequeues jobs. It uses transactional queues to delete jobs only upon successful completion, that allows to process jobs reliably inside ASP.NET applications. MSMQ queues contain only job identifiers, other information is still persisted in the SQL Server database.

Advantages of using MSMQ

No additional latency. It uses blocking calls to fetch jobs – they will be processed as soon as possible.
Immediate re-queueing of terminated jobs. If the processing was terminated in the middle, it will be started again immediately after application restart. SQL Server implementation uses InvisibleTimeout to distinguish between long-running and aborted jobs.

Installation

MSMQ support for SQL Server job storage implementation, like other HangFire extensions, is a NuGet package. So, you can install it using NuGet Package Manager Console window:

PM> Install-Package HangFire.SqlServer.MSMQ

Usage

To use MSMQ queues, you should do the following steps:

Create them manually on each host. Don’t forget to grant appropriate permissions.
Register all MSMQ queues in current SqlServerStorage instance.

var storage = new SqlServerStorage("<connection string>");
storage.UseMsmqQueues(@".\hangfire-{0}", "critical", "default");
// or storage.UseMsmqQueues(@".\hangfire-{0}") if you are using only "default" queue.

JobStorage.Current = storage;

To see the full list of supported paths and queues, check the MSDN article.

Limitations

Only transactional MSMQ queues supported for reability reasons inside ASP.NET.
You can not use both SQL Server Job Queue and MSMQ Job Queue implementations in the same server (see below). This limitation relates to HangFire.Server only. You can still enqueue jobs to whatever queues and watch them both in HangFire.Monitor.

The following case will not work: the critical queue uses MSMQ, and the default queue uses SQL Server to store job queue. In this case job fetcher can not make the right decision.

var storage = new SqlServerStorage("<connection string>");
storage.UseMsmqQueues(@".\hangfire-{0}", "critical");

JobStorage.Current = storage;

var options = new BackgroundJobServerOptions
{
    Queues = new [] { "critical", "default" }
};

var server = new AspNetBackgroundJobServer(options);
server.Start();

Transition to MSMQ queues

If you have a fresh installation, just use the UseMsmqQueues method. Otherwise, your system may contain unprocessed jobs in SQL Server. Since one HangFire.Server instance can not process job from different queues, you should deploy two instances of HangFire.Server, one listens only MSMQ queues, another – only SQL Server queues. When the latter finish its work (you can see this from HangFire.Monitor – your SQL Server queues will be removed), you can remove it safely.

If you are using default queue only, do this:

/* This server will process only SQL Server table queues, i.e. old jobs */

var oldStorage = new SqlServerStorage("<connection string>");
var oldOptions = new BackgroundJobServerOptions
{
    ServerName = "OldQueueServer" // Pass this to differentiate this server from the next one
};

var oldQueueServer = new AspNetBackgroundJobServer(oldOptions, oldStorage);
oldQueueServer.Start();

/* This server will process only MSMQ queues, i.e. new jobs */

// Assign the storage globally, for client, server and monitor.
JobStorage.Current = 
    new SqlServerStorage("<connection string>").UseMsmqQueues(@".\hangfire-{0}");

var server = new AspNetBackgroundJobServer();
server.Start();

If you use multiple queues, do this:

/* This server will process only SQL Server table queues, i.e. old jobs */

var oldStorage = new SqlServerStorage("<connection string>");
var oldOptions = new BackgroundJobServerOptions
{
    Queues = new [] { "critical", "default" }, // Include this line only if you have multiple queues
    ServerName = "OldQueueServer" // Pass this to differentiate this server from the next one
};

var oldQueueServer = new AspNetBackgroundJobServer(oldOptions, oldStorage);
oldQueueServer.Start();

/* This server will process only MSMQ queues, i.e. new jobs */

// Assign the storage globally, for client, server and monitor.
JobStorage.Current = 
    new SqlServerStorage("<connection string>").UseMsmqQueues(@".\hangfire-{0}");

var options = new BackgroundJobServerOptions
{
    Queues = new [] { "critical", "default" }
};

var server = new AspNetBackgroundJobServer(options);
server.Start();

Are your methods ready to run in background?

2014-05-10T00:00:00+00:00

HangFire takes regular classes and regular methods to perform them in the background, because it is simple to use them:

BackgroundJob.Enqueue(() => Console.WriteLine("Hi!"));

This snippet says that the Console.WriteLine method will be called in background. But notice that the name of the method is Enqueue, and not Call, Invoke and so on.

The name of the method was chosen to highlight that invocation of a given method is only being queued in the current execution context and returns the control to a calling thread immediately after enqueueing. It will be invoked in a different execution context.

What does this mean? Several things, that may break your usual expectations about method invocation process. You should be aware of them.

Differences between local and background method invocation

Method invocation is being serialized

Before creating a background job, the information about the given method (its type, method name and parameter types) are serialized to strings. MethodInfo serialization process is absolutely invisible to a user, unlike arguments serialization.

Arguments are also serialized to string, but arguments serialization process uses the TypeConverter class. All standard classes like numbers, strings, dates and so on already have the corresponding TypeConverter implementation, but if you want to pass an instance of a custom class as an argument, you should write the custom converter first.

// Does not work until you implement the custom TypeConverter.
BackgroundJob.Enqueue(() => CheckArticle(new Article()));

Furthermore, serialized arguments can take more space, and it often is more efficient to pass database identifiers or file names instead of their contents.

Execution context is being changed

In the simplest case, such as using ThreadPool.QueueUserWorkItem or Task.Factory.StartNew methods, only thread is being changed. But in HangFire, you can use different process, or different server to process background jobs.

So, the execution context term includes not only thread context, request context and so on, but also static data, including local locks, local filesystem, etc.

That is why if you are querying data inside a background job that corresponds to the execution context where the job was enqueued, it may fail. If you need to pass the current state to a job, use arguments or shared storage.

public void Method()
{
    // Does not work, use distributed locks.
    lock (_object) { /* ... */ } 

    // Does not work either, pass data as an argument.
    if (HttpContext.Current.Request.IsLocal)
    {
        // Processing
    }
}

Delayed invocation

Background job methods are not invoked immediately. It is placed on a queue and waits until any worker pick it up. This leads to an undefined start time and end time.

Undefined start-up time

Your method can be invoked tomorrow, after two weeks or six months (always true for scheduled jobs, but works with “fail-deploy-retry” practice as well). If it is true even for regular method calls, that application data can be changed or arguments can become stale during the method invocation, especially in a highly concurrent web applications, the probability of these situations in background job processing is very high.

Always double-check the data that you pass as arguments and think about its changing nature. Here are some examples:

You want to publish an article tomorrow, think, what do you need to do: publish its current state, or publish the article itself. In the most cases you’d choose to publish the article itself, changed or unchanged. That is why in this case you need to pass an article identifier as an argument.
You want to check comments for spam and it is possible to change them. You are creating a new background job on each edit attempt. In this case, you need to check exactly the given text for each edit, so pass the whole text as an argument. And after the check is completed, you can compare this text with the current one. If it was changed, then don’t do anything.

Undefined end time

We are thinking about the end time, when we want to tell our users about the job was completed. If you do something inside the request processing synchronously, you can rely on that fact that this information will be available for a user immediately. That is why you can redirect her to a just created article page.

But when, for example, you are creating the same article in a background job, you can redirect user to a non-existing page yet, because you can not guarantee that the enqueued job will be processed in time.

Of course, you always can tune your system to perform background jobs as soon as possible, but you can not eliminate the delay between enqueue time and a real invocation completely.

So you need to think how to show yet unprocessed entities (with loaders, progress bars and so on), and how to report the job completion. For a latter task, you can always use polling, server push (for example, using SignalR), or force users to reload the page manually.

Delayed failure

If you pass user input as an argument of your job, you should validate them first. Otherwise the job will always fail, retry, fail again, retry again and so on. It will never be completed without manual intervention.

You can disable automatic retry on failure, but there is another problem – since the immediate invocation is not guaranteed (see the section Delayed invocation), your user may leave the site and she will not be able to correct the data.

So, validate your arguments early, and instead of doing this:

public ActionResult CreateComment(string message)
{
    BackgroundJob.Enqueue(() => CheckComment(message));
    ...
}

public void CheckComment(string message)
{
    if (message == null) 
    {
        throw new ArgumentNullException("message");
    }

    // Processing
}

Do this:

public ActionResult CreateComment(string message)
{
    if (message == null)
    {
        return ValidationError(message); // Pseudo-code
    }
    BackgroundJob.Enqueue(() => CheckComment(message));
    ...
}

public void CheckComment(string message)
{
    // But you can leave the guard condition here.
    // Processing
}

Method can be called multiple times

Your method can be retried manually (through the Monitor interface) or automatically on failure (i.e on unexpected exception that is unhandler by the method itself).

So, be prepared for this situation. Try to make all your background job methods idempotent. Be prepared, in very rare cases, that your background job can fire multiple times.

You can always disable the automatic retry feature by applying the [Retry(0)] filter to the exact method or globally. But to successfully fight with ASP.NET unexpected application domain unload in the middle of a job processing, HangFire retries them automatically despite of the given attribute. But don’t worry too much, these cases happen very rarely.

But as a general rule remember, that your job will be performed at least once. You can test your job for idempotence by calling it multiple times and compare the result:

public void TestIdempotence()
{
   // Arrange
   // ...

   // Act
   YourMethod();
   YourMethod();

   // Assert
   // For example, check that records were not duplicated.
}

Method may become unavailable

Your application can be redeployed with a different code base after a background job was enqueued. That is why you should change the signature of your background job methods carefully.

You can safely change parameter names, but the following things will lead to a broken job (however you are able to fix it and re-deploy the application):

Addition and removal of method parameters.
Parameter type changes.
Parameter reordering.
Method name changes.
Method’s type name changes.
Method removal.
Method’s type removal.
Type’s namespace changes.
Type’s assembly changes.

Of course you can do all the above things, if there are no jobs in a storage. But instead of doing this, add a new method without touching the old one and call the new method from the old one until all old jobs become processed:

public void OldMethod(string arg1, int arg2) 
{ 
    // Redirect
    NewMethod(arg1, arg2, DefaultValueForArg3);
}

public void NewMethod(string arg1, int arg2, double arg3)
{
    // Real processing
}

Summary

There are a lot of differences between local and background method invocation, but you likely know the most of them, because they relate to asynchronous and concurrent programming as well.

If you have any questions, ask them in the comments form below the post or start a new topic on http://discuss.hangfire.io if you want to share code snippets.

HangFire 0.8 Released

2014-05-02T00:00:00+00:00

Today I’ve pushed the next version of HangFire project that allows to process background jobs inside ASP.NET applications in a reliable way.

Although this release contains only one new feature that is visible for all users (security improvements), it brings HangFire closer than ever to the version 1.0. But let’s see the new changes.

Note. The conceptual part of the following information is correct, but implementation details changed since version 1.0. Please, see the documentation instead of relying this information.

Remote requests are denied by default

HangFire has integrated monitoring system that shows you information about background jobs and acts as an ASP.NET’s IHttpHandler. Registration is performed by automatically changing the web.config file during the installation of the HangFire.Web package (it is a dependency of HangFire bootstrap package) by NuGet Package Manager.

This HTTP handler displays the internals of your application. Furthermore it offers to interact with this internal state: background jobs can be retried, possibly removed, scheduled earlier than necessary and so on. What if it become available for malicious users? It is hard to guess the actual consequences for a wide range of applications, but it is simple to understand that this behavior is highly undesirable.

The automatics greatly simplifies the installation process, but performs it on the quiet, making it simple to overlook that Monitor handler is ever exist or requires additional set-up. So, it is required to provide strong rules that work by default.

To solve this problem, we need to authorize users. ASP.NET applications have authorization feature by default, that is based on user names or their roles. But it is impossible to guess what user names and roles uses your ASP.NET application (if your application uses them at all). So, we need more simple solution.

The simplest solution is to block all remote requests by default using the Request.IsLocal property. Local requests are accepted as usually, and this decision does not bring problems for application developers. Another libraries, such as Elmah or Glimpse are also using this authorization policy by default.

The policy can be configured from the appSettings section of web.config file – I don’t want to introduce separate configuration section for a single setting.

Disable remote access to HangFire Monitor (by default):

<appSettings>
  <!-- You can also remove this line to keep the default settings -->
  <add key="hangfire:EnableRemoteMonitorAccess" value="false"/>
  ...
</appSettings>

Enable remote access to HangFire Monitor:

<appSettings>
  <add key="hangfire:EnableRemoteMonitorAccess" value="true"/>
  ...
</appSettings>

Granting or denying access via ASP.NET authorization

Remote Monitor access policy is good. But once your application is deployed to production environment, you need to be able to access Monitor, otherwise it is useless.

Elmah library author and contributors conducted a deep investigation on a topic and proposed the solution to provide use ASP.NET authorization for such registered handlers:

The handler registrations need to be moved under the location tag. Having them outside does not secure access sufficiently.
The location element’s path attribute carries the same value as the path attribute of the handler registrations. Unfortunately, it has to be repeated so if you change one, the others must be updated to match.
The authorization element is where the authorization rules go and where you will want to selectively grant access.

So, starting from this version, HangFire Monitor HTTP handler is being registered under the location tag:

<location path="hangfire.axd" inheritInChildApplications="false">
  <system.web>
    <authorization>
      <deny users="*" />  
    </authorization>
  </system.web>
  <system.webServer>
    <handlers>
      <add name="HangFire" path="hangfire.axd" verb="*" type="HangFire.Web.HangFirePageFactory, HangFire.Web" />
    </handlers>
  </system.webServer>
</location>

The configuration example above denies access to all users, but that is a good starting point. You will probably want to add rules that allow access to only specific users and/or roles. For example, you might have a role for administrators and developers called admin and dev, respectively. To allow users that are members of either role, you could configure the authorization section above as follows:

<authorization>  
  <allow roles="admin" />  
  <allow roles="dev" />  
  <deny users="*" />  
</authorization>

But to enable these rules, you need to set-up ASP.NET Authentication using, for example, the new and shining ASP.NET Identity project.

Moving closer to 1.0

This version contains 300+ new unit tests, their total number is 550, and the test coverage of HangFire.Core.dll is 99%. Although this does not say that HangFire is 100% bug-free, this is an indicator of the fact that the most code is tested and it is fully unit testable. New features can be added much faster. To do this, I’ve completely rewritten the Server subsystem that resulted in a more clean application model.

Full unit testability was one of the goals I wanted to achieve, and the fact that it requires a lot of breaking changes stopped me even to think about production-ready version. I want to use semantic versioning and do not want to have HangFire 28.0 in a few months.

Now I’ve started to think about production readiness, but I need a lot of feedback to give any stability guarantees.

Recently I’ve launched a new forum for HangFire to replace unnoying (for me) Google Groups – http://discuss.hangfire.io. It is modern, clean and easy to use, based on Discourse. So, have ideas or problems? Let’s discuss!

odinserj

Hangfire Blog Moved

https://www.hangfire.io/blog

Hangfire 1.3.0 Released

Logging that just works

Debugging with Symbolsource

Changes

Links

Introducing Hangfire Pro

The Destiny of Hangfire.Redis

Bad News

Good News

Hangfire Pro Today

Hangfire Pro Roadmap

Batch Jobs

Async Methods Support

What are you waiting for?

Hangfire 1.0 Released

Release Notes

Upgrading from 0.x

Breaking Changes

Links

HangFire 0.9 Released

Recurring jobs

Changes

Upgrading

Links

HangFire 0.8.3 Released

Cancellation tokens

Hanging jobs

TL;DR

Changes

Links

HangFire 0.8.2 Released

Batch operations

Additional metrics

DisableConcurrentExecution filter

Changes

Links

HangFire 0.8.1 released

Release notes

Deleting jobs

Usage

Manual deletion

MSMQ support for SQL Server storage

Advantages of using MSMQ

Installation

Usage

Limitations

Transition to MSMQ queues

Are your methods ready to run in background?

Differences between local and background method invocation

Method invocation is being serialized

Execution context is being changed

Delayed invocation

Undefined start-up time

Undefined end time

Delayed failure

Method can be called multiple times

Method may become unavailable

Summary

HangFire 0.8 Released

Remote requests are denied by default

Granting or denying access via ASP.NET authorization

Moving closer to 1.0