mongodb.driverのソースコードを読んでみました

障害調査でmongodb.driverのソースコードを読む機会があったのでブログ化してみました。

1.前提
テスト環境で3台構成でレプリカセットを組んでいるうちの2台(SECONDARY,ARBITAR)を運用コスト削減の為に通常はシャットダウン。
PRIMARYをレプリカセットを解除してスタンドアローンで通常運用。
月次のWindowsパッチ当ての時のみ3台構成のレプリカセットに戻し、同期させ、作業終了後にレプリカセットを解除して2台を元通りに
シャットダウンという運用をしていました。
レプリカ構成からスタンドアローン構成に変更する時はlocal dbのDropは行わず、有効期限の古いドキュメントが削除できない旨のWARNINGメッセージが
ログイン時に表示されるようにしていました。
WEBサーバ側のmongoURI指定はどちらの構成でも共用できるようにPrimary,Secondaryの2ホストを指定していました。

2.事象
WEBサーバが表示不可になり、
ログには、
「Unable to connect to a member of the replica set matching the read preference PRIMARY」
とスタンドアローン構成のはずのmongodbなのにレプリカセットのPRIMARYに接続できない旨のエラーメッセージが出力されていました。
IISの再起動とサーバ再起動で復旧したものの、翌営業日に事象が再発。

3.対処
WEBサーバ側指定のmongoURIに複数ホストを指定しているのが怪しいとみて、
スタンドアローン構成時には1ホストのみ指定、
レプリカ構成時にはPrimary,Secondaryの2ホストを指定
するようにconfigを切り替えて使用するようにしたところ、
事象の再発は無くなりました。

4.原因
不明

5.mongodb.driverのソースコードを読んでわかったこと
mongoURIに
レプリカセット名が設定されていれば、ReplicaSetMongoServerProxyの処理へ
サーバ数が1ならば、DirectMongoServerProxyの処理へ
ShardRouterなら、ShardedMongoServerProxyの処理へ
それ以外は、DiscoveringMongoServerProxyの処理へ

障害時の
「Unable to connect to a member of the replica set matching the read preference Primary」
のエラーメッセージが出力されるのは、
ReplicaSetMongoServerProxy,ShardedMongoServerProxyの処理だけなので
1ホストだけの記述にすることで、該当エラーメッセージが出力されることは無くなります。

障害時には
DiscoveringMongoServerProxyの処理を経て
ReplicaSetMongoServerProxy,ShardedMongoServerProxyのいずれかの処理に入ったと予想されますが、
ReplicaSetMongoServerProxyの処理に入るには、
mongodbのステータスチェックコマンド(db.isMaster())の返り値に
レプリカセットの状態が出力される必要がありますが、
コマンドを試行してもStandaloneの出力でした。

何故、当該エラーが出力されたのかは不明でした。

6.ソースコードを読んだ内容

\mongo-csharp-driver-1.10.0\MongoDB.Driver\Communication\Proxies\MongoServerProxyFactory.cs

namespace MongoDB.Driver.Internal
{
/// <summary>
/// Creates a MongoServerInstanceManager based on the settings.
/// </summary>
internal class MongoServerProxyFactory
{
// public methods
/// <summary>
/// Creates an IMongoServerProxy of some type that depends on the server settings.
/// </summary>
/// <param name=”settings”>The settings.</param>
/// <returns>An IMongoServerProxy.</returns>
public IMongoServerProxy Create(MongoServerSettings settings)
{
var connectionMode = settings.ConnectionMode;
if (settings.ConnectionMode == ConnectionMode.Automatic)
{
if (settings.ReplicaSetName != null)
{
connectionMode = ConnectionMode.ReplicaSet;
}
else if (settings.Servers.Count() == 1)
{
connectionMode = ConnectionMode.Direct;
}
}

switch (connectionMode)
{
case ConnectionMode.Direct:
return new DirectMongoServerProxy(settings);
case ConnectionMode.ReplicaSet:
return new ReplicaSetMongoServerProxy(settings);
case ConnectionMode.ShardRouter:
return new ShardedMongoServerProxy(settings);
default:
return new DiscoveringMongoServerProxy(settings);
}
}
}
}

レプリカセット名が設定されていれば、ReplicaSetMongoServerProxyの処理へ
サーバ数が1ならば、DirectMongoServerProxyの処理へ
ShardRouterなら、ShardedMongoServerProxyの処理へ
それ以外は、DiscoveringMongoServerProxyの処理へ

1ホストだけかどうかで処理が分岐しているので、config修正の効果はあり。

#複数ホスト且つレプリカ指定をせずにを書いた場合の処理
\mongo-csharp-driver-1.10.0\MongoDB.Driver\Communication\Proxies\DiscoveringMongoServerProxy.cs

private void CreateActualProxy(MongoServerInstance instance, BlockingQueue<MongoServerInstance> connectionQueue)
{
lock (_lock)
{
if (instance.InstanceType == MongoServerInstanceType.ReplicaSetMember)
{
_serverProxy = new ReplicaSetMongoServerProxy(_settings, _instances, connectionQueue, _connectionAttempt);
}
else if (instance.InstanceType == MongoServerInstanceType.ShardRouter)
{
_serverProxy = new ShardedMongoServerProxy(_settings, _instances, connectionQueue, _connectionAttempt);
}
else if (instance.InstanceType == MongoServerInstanceType.StandAlone)
{
var otherInstances = _instances.Where(x => x != instance).ToList();
foreach (var otherInstance in otherInstances)
{
otherInstance.Disconnect();
}

_serverProxy = new DirectMongoServerProxy(_settings, instance, _connectionAttempt);
}
else
{
throw new MongoConnectionException(“The type of servers in the host list could not be determined.”);
}
}
}

\mongo-csharp-driver-1.10.0\MongoDB.Driver\Communication\MongoServerInstanceType.cs

namespace MongoDB.Driver
{
/// <summary>
/// Represents an instance of a MongoDB server host (in the case of a replica set a MongoServer uses multiple MongoServerInstances).
/// </summary>
public enum MongoServerInstanceType
{
/// <summary>
/// The server instance type is unknown. This is the default.
/// </summary>
Unknown,
/// <summary>
/// The server is a standalone instance.
/// </summary>
StandAlone,
/// <summary>
/// The server is a replica set member.
/// </summary>
ReplicaSetMember,
/// <summary>
/// The server is a shard router (mongos).
/// </summary>
ShardRouter
}
}

mongo-csharp-driver-1.10.0\MongoDB.Driver\Communication\Proxies\MultipleInstanceMongoServerProxy.cs(508,9) [UTF-8]:
エラーメッセージはこのファイルに記述されている。
MultipleInstanceMongoServerProxy.cs の内容が使用されているのは、
ReplicaSetMongoServerProxy.cs
ShardedMongoServerProxy.cs
のみ

private readonly ConnectedInstanceCollection _connectedInstances;

public MongoServerInstance ChooseServerInstance(ReadPreference readPreference)
{
for (int attempt = 1; attempt <= 2; attempt++)
{
if (!_connectedInstances.AreAllServersVersionCompatible)
{
var message = “This version of the driver is not compatible with at least one of the servers.”;
throw new MongoConnectionException(message);
}

var instance = ChooseServerInstance(_connectedInstances, readPreference);
if (instance != null)
{
return instance;
}
if (attempt == 1)
{
Connect(_settings.ConnectTimeout, readPreference);
}
}

throw new MongoConnectionException(“Unable to choose a server instance.”);
}

public void Connect(TimeSpan timeout, ReadPreference readPreference)
{
var timeoutAt = DateTime.UtcNow + timeout;
while (DateTime.UtcNow < timeoutAt)
{
if (ChooseServerInstance(_connectedInstances, readPreference) != null)
{
return;
}

if (Interlocked.CompareExchange(ref _outstandingInstanceConnections, 0, 0) > 0)
{
Thread.Sleep(TimeSpan.FromMilliseconds(20));
continue;
}

lock (_lock)
{
// test this again (kinda like the double lock check pattern). This value may
// be different and we don’t want to issue another round of connects needlessly.
if (Interlocked.CompareExchange(ref _outstandingInstanceConnections, 0, 0) > 0)
{
Thread.Sleep(TimeSpan.FromMilliseconds(20));
continue;
}

// if we are already fully connected and an instance still isn’t chosen,
// then one simply doesn’t exist, so we’ll break immediately and throw a
// connection exception.
if (_state == MongoServerState.Connected)
{
break;
}

_state = MongoServerState.Connecting;
_connectionAttempt++;

foreach (var instance in _instances)
{
ConnectInstance(instance);
}
}
}

ThrowConnectionException(readPreference);
}

private void ThrowConnectionException(ReadPreference readPreference)
{
List<Exception> exceptions;
lock (_lock)
{
exceptions = _instances.Select(x => x.ConnectException).Where(x => x != null).ToList();
}
var firstException = exceptions.FirstOrDefault();
string message;

if (firstException == null)
{
message = string.Format(“Unable to connect to a member of the replica set matching the read preference {0}”, readPreference);
}
else
{
message = string.Format(“Unable to connect to a member of the replica set matching the read preference {0}: {1}.”, readPreference,

firstException.Message);
}
var connectionException = new MongoConnectionException(message, firstException);
connectionException.Data.Add(“InnerExceptions”, exceptions); // useful when there is more than one
throw connectionException;
}

□検索条件 “MultipleInstanceMongoServerProxy”
検索対象 *.*
フォルダ C:\mongo-csharp-driver-1.10.0\MongoDB.Driver
(サブフォルダも検索)
(英大文字小文字を区別しない)
(文字コードセットの自動判別)
(一致した行を出力)

\mongo-csharp-driver-1.10.0\MongoDB.Driver\MongoDB.Driver.csproj(381,45) [UTF-8]: <Compile Include=”Communication\Proxies\MultipleInstanceMongoServerProxy.cs” />
\mongo-csharp-driver-1.10.0\MongoDB.Driver\Communication\Proxies\MultipleInstanceMongoServerProxy.cs(27,29) [UTF-8]: internal abstract class

MultipleInstanceMongoServerProxy : IMongoServerProxy
\mongo-csharp-driver-1.10.0\MongoDB.Driver\Communication\Proxies\MultipleInstanceMongoServerProxy.cs(39,58) [UTF-8]: /// Initializes a new instance of the <see

cref=”MultipleInstanceMongoServerProxy”/> class.
\mongo-csharp-driver-1.10.0\MongoDB.Driver\Communication\Proxies\MultipleInstanceMongoServerProxy.cs(42,19) [UTF-8]: protected MultipleInstanceMongoServerProxy

(MongoServerSettings settings)
\mongo-csharp-driver-1.10.0\MongoDB.Driver\Communication\Proxies\MultipleInstanceMongoServerProxy.cs(52,58) [UTF-8]: /// Initializes a new instance of the <see

cref=”MultipleInstanceMongoServerProxy”/> class.
\mongo-csharp-driver-1.10.0\MongoDB.Driver\Communication\Proxies\MultipleInstanceMongoServerProxy.cs(59,19) [UTF-8]: protected MultipleInstanceMongoServerProxy

(MongoServerSettings settings, IEnumerable<MongoServerInstance> instances, BlockingQueue<MongoServerInstance> connectionQueue, int connectionAttempt)
mongo-csharp-driver-1.10.0\MongoDB.Driver\Communication\Proxies\ReplicaSetMongoServerProxy.cs(26,56) [UTF-8]: internal sealed class ReplicaSetMongoServerProxy :

MultipleInstanceMongoServerProxy
\mongo-csharp-driver-1.10.0\MongoDB.Driver\Communication\Proxies\ShardedMongoServerProxy.cs(25,53) [UTF-8]: internal sealed class ShardedMongoServerProxy :

MultipleInstanceMongoServerProxy

\mongo-csharp-driver-1.10.0\MongoDB.Driver\MongoClientSettings.cs(478,28) [UTF-8]:
478行目
clientSettings.ReadPreference = (builder.ReadPreference == null) ? ReadPreference.Primary : builder.ReadPreference.Clone();

ReadPreference指定が無ければ、Primaryを格納。

\mongo-csharp-driver-1.10.0\MongoDB.Driver\ReadPreference.cs(27,17) [UTF-8]:
27行目

public enum ReadPreferenceMode
{
/// <summary>
/// Use primary only.
/// </summary>
Primary,
/// <summary>
/// Use primary if possible, otherwise a secondary.
/// </summary>
PrimaryPreferred,
/// <summary>
/// Use secondary only.
/// </summary>
Secondary,
/// <summary>
/// Use a secondary if possible, otherwise primary.
/// </summary>
SecondaryPreferred,
/// <summary>
/// Use any near by server, primary or secondary.
/// </summary>
Nearest
}

// constructors
/// <summary>
/// Initializes a new instance of the ReadPreference class.
/// </summary>
public ReadPreference()
: this(ReadPreferenceMode.Primary)
{
}

\mongo-csharp-driver-1.10.0\MongoDB.Driver\Communication\MongoServerInstance.cs

// private methods
private void LookupServerInformation(MongoConnection connection)
{
IsMasterResult isMasterResult = null;
bool ok = false;
try
{
var isMasterCommand = new CommandDocument(“ismaster”, 1);
isMasterResult = RunCommandAs<IsMasterResult>(connection, “admin”, isMasterCommand);

MongoServerBuildInfo buildInfo;
try
{
var buildInfoCommand = new CommandDocument(“buildinfo”, 1);
var buildInfoResult = RunCommandAs<CommandResult>(connection, “admin”, buildInfoCommand);
buildInfo = MongoServerBuildInfo.FromCommandResult(buildInfoResult);
}
catch (MongoCommandException ex)
{
// short term fix: if buildInfo fails due to auth we don’t know the server version; see CSHARP-324
if (ex.ErrorMessage != “need to login”)
{
throw;
}
buildInfo = null;
}

ReplicaSetInformation replicaSetInformation = null;
MongoServerInstanceType instanceType = MongoServerInstanceType.StandAlone;
if (isMasterResult.IsReplicaSet)
{
var peers = isMasterResult.Hosts.Concat(isMasterResult.Passives).Concat(isMasterResult.Arbiters).ToList();
replicaSetInformation = new ReplicaSetInformation(isMasterResult.ReplicaSetName, isMasterResult.Primary, peers, isMasterResult.Tags,

isMasterResult.ReplicaSetConfigVersion);
instanceType = MongoServerInstanceType.ReplicaSetMember;
}
else if (isMasterResult.Message != null && isMasterResult.Message == “isdbgrid”)
{
instanceType = MongoServerInstanceType.ShardRouter;
}

var featureContext = new FeatureContext
{
BuildInfo = buildInfo,
Connection = connection,
IsMasterResult = isMasterResult,
ServerInstanceType = instanceType
};
var featureSet = new FeatureSetDetector().DetectFeatureSet(featureContext);

var newServerInfo = new ServerInformation
{
BuildInfo = buildInfo,
FeatureSet = featureSet,
InstanceType = instanceType,
IsArbiter = isMasterResult.IsArbiterOnly,
IsMasterResult = isMasterResult,
IsPassive = isMasterResult.IsPassive,
IsPrimary = isMasterResult.IsPrimary,
IsSecondary = isMasterResult.IsSecondary,
MaxBatchCount = isMasterResult.MaxWriteBatchSize,
MaxDocumentSize = isMasterResult.MaxBsonObjectSize,
MaxMessageLength = isMasterResult.MaxMessageLength,
ReplicaSetInformation = replicaSetInformation
};
MongoServerState currentState;
lock (_serverInstanceLock)
{
currentState = _state;
}
SetState(currentState, newServerInfo);
ok = true;
}
finally
{
if (!ok)
{
ServerInformation currentServerInfo;
lock (_serverInstanceLock)
{
currentServerInfo = _serverInfo;
}

// keep the current instance type, build info, and replica set info
// as these aren’t relevent to state and are likely still correct.
var newServerInfo = new ServerInformation
{
BuildInfo = currentServerInfo.BuildInfo,
FeatureSet = currentServerInfo.FeatureSet,
InstanceType = currentServerInfo.InstanceType,
IsArbiter = false,
IsMasterResult = isMasterResult,
IsPassive = false,
IsPrimary = false,
IsSecondary = false,
MaxDocumentSize = currentServerInfo.MaxDocumentSize,
MaxMessageLength = currentServerInfo.MaxMessageLength,
MaxBatchCount = currentServerInfo.MaxBatchCount,
ReplicaSetInformation = currentServerInfo.ReplicaSetInformation
};

SetState(MongoServerState.Disconnected, newServerInfo);
}
}
}

admindb で db.isMaster()コマンドを発行して、mongoインスタンスの情報を取得

コマンド実行結果

> db.isMaster()
{
“ismaster” : true,
“maxBsonObjectSize” : 16777216,
“maxMessageSizeBytes” : 48000000,
“maxWriteBatchSize” : 1000,
“localTime” : ISODate(“2019-03-27T05:02:54.530Z”),
“maxWireVersion” : 4,
“minWireVersion” : 0,
“ok” : 1
}

>

Standalone 判定される返り値になっている。

Comments are closed.