We fixed the issue that caused messages to get stuck in the transport queues of the local exchange server 2016 and exchange server 2019. the issue is related to a date check failure with the change of the new year and not a failure of the av motor itself. this is not a problem with the malware scan or the malware engine, and it is not a security-related problem. the version check performed on the signature file is causing the malware engine to crash, causing messages to get stuck in transport queues.
We have now created a fix to address the issue of messages getting stuck in transport queues on exchange server 2016 and exchange server 2019 due to a latent date issue in a signature file used by the malware scanning engine within from exchange servers. client action is required to implement this fix. when the issue occurs, you will see errors in the application event log on the exchange server, specifically events 5300 and 1106 (fipfs), as illustrated below:
using the automated solution
- download the script here: https://aka.ms/resetscanengineversion
- before running the script, change the execution policy for powershell scripts by running set-executionpolicy -executionpolicy remotesigned .
- Run the script on each exchange mailbox server that downloads antimalware updates in your organization (use an elevated exchange management shell).
Edge Transport servers are not affected by this issue. you can run this script on multiple servers in parallel. once the script has completed, you will see the following output:
using the manual solution
Instead of using the script, customers can also manually perform steps to resolve the issue and restore service. To manually resolve this issue, you must perform the following steps on each Exchange Mailbox server in your organization that downloads antimalware updates. Edge Transport servers are not affected by this issue.
verify that the affected version is installed run get-engineupdateinformation and check the updated version information. if it starts with “22…” then proceed. if the installed version starts with “21…”, you do not need to take any action.
remove existing engine and metadata1. stop the microsoft filtering management service. When prompted to also stop the microsoft exchange transport service, click yes.2. use task manager to make sure updateservice.exe is not running.3. delete the following folder: %programfiles%microsoftexchange serverv15fip-fsdataenginesamd64microsoft.4. delete all files in the following folder: %programfiles%microsoftexchange serverv15fip-fsdataenginesmetadata.
update to the latest engine1. start the microsoft filtering management service and the microsoft exchange transport service.2. open the exchange management shell, go to the scripts folder (%programfiles%microsoftexchange serverv15scripts) and run update-malwarefilteringserver.ps1 <server fqdn>.
verify engine update information1. in the exchange management shell, run add-pssnapin microsoft.forefront.filtering.management.powershell.2. run get-engineupdateinformation and verify that the updated version information is 2112330001 (or higher)
after upgrading the engine, we also recommend that you verify that the mail flow is working and that the fipfs error events are not present in the application event log.
frequently asked questions
I’m not sure if this issue affects my organization. how do i find out?run the latest version of the healthchecker script (https://aka.ms/exchangehealthchecker) on all exchange servers in your organization and look for the fip-fs warning that will be displayed if your the server is affected and further action is required.
Is the fix for this issue automated?Implementing the fix requires customer action, and it will take some time to make the necessary changes, download the updated files, and clear the transport queues. actions can be automated with the scan engine reset script from https://aka.ms/resetscanengineversion or can be performed manually. Whether you perform the steps automatically or manually, they must be performed on all local Exchange 2016 and Exchange 2019 servers in your organization. if you use the automated script, you can run it on multiple servers in parallel.
How long will it take to run the automated script?Depending on the size of your organization, the script may take some time to run; be patient.
How long will it take to clean up the queues after the script has run?Depending on the number of messages that were queued and the number of new messages that the transport needs to process, the time may vary so please be patient and check that the queues are running low (the number of messages is decreasing) using the get-queue command.
We are in a hybrid exchange environment. what should we do?if you’re using your local exchange server to send email (for example, using a centralized mail flow or sending messages from local devices), follow this blog post and use the script to switch the settings on your local servers used for email transport. If you’re using on-premises exchange just for managing exchange recipients, you don’t need to take any action.
what services does the script stop?the following services will be restarted: microsoft filtering management and microsoft exchange transport.
We have temporarily disabled antimalware. should it be enabled after following this blog post?if you have temporarily disabled the antimalware service, you need to enable it after following this blog post (use the enable-antimalwarescanning.ps1 script). the solution described in this post is a complete fix for this issue and will result in the transport queues being cleared and the antimalware engine working as expected.
the updated scanning engine version starts with 2112330001 (or higher); is this correct? Should we be concerned that it appears to refer to a date that doesn’t exist?Microsoft fully supports the recently updated scan engine. while we need to work on this longer-term stream, the scan engine version was not rolled back, but implemented in this new stream. the scanning engine will continue to receive updates in this new stream.
what if my exchange servers don’t have access to the internet?if your exchange mail servers don’t download anti-malware updates from the internet, you don’t need to take any manual action. in that case, the servers haven’t been downloading antimalware updates to begin with, and the problem described here won’t exist.
We have a 2013 exchange server, and although there are no crashes, I see that the server has a problematic engine version starting with “22…”. what should we do?exchange server 2013 is not affected by transport failures, so there will be no backlog of email in transport queues. If your exchange 2013 server took the antimalware update and is now on a version that starts with “22…”, you should use the automatic or manual steps in this blog post to get your server on a “21..” engine version. .” to continue receiving antimalware updates. update 1/11/2022: exchange 2013 clients: before running the script, check if the engine was automatically updated to version 21… there is an additional mechanism that updates the engine to the latest version and, for now, its exchange 2013 servers should have automatically fixed the version.
The script fails with the error “Warning: The update request cannot be processed because the engine metadata is not available.” trying to sync” metadata. try running the cmdlet again later.for exchange servers that access the internet via proxy:
- start exchange management shell
- add-pssnapin microsoft.forefront.filtering.management.powershell
- set-proxysettings -enabled $true -server <proxyserver> ; -port <proxy.port>
- rerun script
if still not resolved:
- copy msvcr110.dll from c:windowssystem32 to %programfiles%microsoftexchange serverv15fip-fsbin and %programfiles%microsoftexchange serverv15bin
- restart the exchange server
- rerun the script
I have many exchange servers in my environment; Is there a way to distribute the definition files locally?
- Follow the steps in this article and download the update files to a server. example: update-engines.ps1 -enginedirpath c:scanengineupdates
- share the folder where the update files were copied, for example \server1amware
- run the the following command on all remaining servers that need to copy the update files: set-malwarefilteringserver -primaryupdatepath \server1amware -identity mail1.contoso.com
- re-run the scan engine restart script on all servers.
Important changes to this blog post:
- 1/1 – Added more information to exchange 2013 FAQ on automatic engine update
- 1/4 – Added FAQ on how to find affected servers using script status check
- 1/3 – Added an FAQ for local distribution of updates within the organization
- 1/3 – Added an FAQ related to configuration proxy and script error
- 1/3 – Added FAQ for exchange servers 2013
- 1/3 – Add add-pssnapin to automated process ; various other minor changes
- 1/2: Clarified the exact process for running the automated script solution; various other minor changes and clarifications
- 1/2 – Added FAQ section to blog post
- 1/1 – Major update mentioning our manual and scripted solution for this problem; removed disabling antimalware service as a workaround
- 1/1: original version
the exchange team